13 ms·

DevOps is a culture, not a role

236 points by mromnia 9y ago

elnygren 9y ago
Yes yes, we get it, it's a mindset not a role. Now could we hire a DevOps guy to setup that CI/CD pipeline, ChatOps stuff, Kubernetes and dev/staging/prod flow so the coders can ship. ;) The original thought was beautiful but the way I see it: DevOps is a branch where you specialise. After school people might learn front/back/db stuff and then specialise to "DevOps stuff" -- or then you have more sysops kind of people who start to venture in the land of automating their work. Both are needed imho.
- enraged_camel 9y ago
  >>ChatOps stuff That is so 2013! I think you meant SlackOps. ;)
- nraynaud 9y ago
  I hate specialization, and I want the person who develop something to think about how that stuff will be deployed and kept online. In particular, some tradeoffs have to be made between the development side and the production side, and I want the same person to see both side and decide with a technical criterion (or just choose one option, and later reverse course and change it), not on a "it's not my problem" criterion.
  - preordained 9y ago
    There is too much. It's specialization out of necessity. I know enough "DevOps" to be dangerous, but in addition to having a deep knowledge of our product's codebase, the multiple features of our product, configurations, etc. there are not enough hours in the day to master (not simply be familiar with) Jenkins pipeline, Docker, AWS, Ansible, Zookeeper, etc, etc. 99% of the people who could claim to have knowledge of all the above in their organization either have a very shallow (read: not useful) knowledge, OR their product itself is simply the gluing together of various DevOps technologies listed, so there is little more to know outside that.
    - cat199 9y ago
      it is definitely possible to have fairly deep level of knowledge at all layers it just takes effort. That said, the larger the project, the more the need for specialization and the less possible it is for someone to keep up with all changes and be a domain expert.
      - aNoob7000 9y ago
        I disagree. Maybe, there's a couple of guys/gals around that can go up and down the software and hardware stack and know where issues may lie when there's a problem, but I doubt there are many. As a someone that's been in the tech industry a very long time, I came to the realization a long time ago that I couldn't be the expert at everything. Can I dabble in Oracle, Windows, Linux? Sure, would I want to bet my job on how much I know some of these products in the time of crisis, hell No!!!
        nraynaud 9y ago
        My last job as an employee, I had to install AWS, develop some server software, develop some web facing software, develop some embedded software (with a bit of easy real time), get the freaking 2G to work, get the sensors to work properly, model the mechanical parts to install the embedded device on site, order all that crap (electronics and mechanics) from the providers, create the boxes, try to think about lightnings and how not to fry the customer's equipment. And get the sandcastle to stay put. That's what happens when you work at startups, there is no plethora of employees, and a lot of work.
ciguy 9y ago
As a DevOps consultant, I think it can actually be both. Culture is central to DevOps but someone has to lead. Sometimes this can be the CTO but often it helps to have a dedicated advocate, especially within larger orgs/teams. Essentially it boils down to semantics. Often DevOps engineers are one part evangelist and one part automation engineer. The software industry has just chosen to call those people "DevOps".
holydude 9y ago
I do not know. We always had "DevOPS" aka sysadmin that knew C,perl and bunch of others. Then it became too much for a single person to know everything and we invented "more ops" and "more programmer". Now we have better tools albeit more complex and we have decided that we again can have this "DevOPS" person whatever that means. But again we are facing this dilemma when there are literally thousands of different tools and ecosystems and you just simply cannot be an EXPERT or even ADVANCED in most of them. We should clearly define what we expect from the role and the candidate. We can't waste other people's time by hiring for DevOPS when we clearly want a PowerShell programmer,windows administrator and database admin in a single package
- dozzie 9y ago
  > But again we are facing this dilemma when there are literally thousands of different tools and ecosystems and you just simply cannot be an EXPERT or even ADVANCED in most of them. Though there are only a handful of ideas and principles, and all of these thousands of different tools are just reimplementations of some of them, and usually not too fancy at that.
  - holydude 9y ago
    That's correct but you can't learn all the nuances and specifics of every tool there is.
    - dozzie 9y ago
      Of course you don't, but those nuances and specifics are usually superficial and don't change how you approach your environment. You think there is that much of a difference between running Puppet, CFEngine, or Chef?
      - holydude 9y ago
        They are not superficial and they definitely change how you approach your environment. I have worked extensively with both CFEngine and Puppet and I can assure you that they have enough differences especially when you consider large heterogenous environments that in no way you can consider using one as a substitute for the other.
benjaminwootton 9y ago
Having worked on over 100 DevOps initiatives, I'm inclined to disagree. 8 years into DevOps, automation, cloud computing etc, it's time to accept that DevOps is a specialism and a function that companies - especially big ones - need to put into place in order to adopt the principles. See here for further thoughts - https://devops.com/im-happy-devops-engineer-job-title/ https://devops.com/im-happy-devops-engineer-job-title/
- signal11 9y ago
  I don't really care about titles so if calling someone a DevOps Engineer helps them and connects the right engineers to the right roles, great. Fighting against this use of "DevOps" would be like fighting against how the word "literally" is misused to mean "figuratively". However for me, "DevOps" is more than just CI/CD and tech automation, it's about not working in silos and having empathy for others in the team (and the customer!). This is a culture and mindset that everyone in the team needs to have.
- inopinatus 9y ago
  I disagree with your disagreement. There's nothing we do under the umbrella of "DevOps" that wasn't part of competent service ownership twenty years ago. I read your article and the point that stood out for me was the notion that the skills to drive tools like chef, puppet are somehow new and different. But I know better, because although I use those tools today, back in the day we used cfengine and a bunch of declaratively/injectively configured shell & perl scripts to achieve basically exactly the same effect. Back then we called this "systems administration". But we were most definitely working with developers and running the SDLC across silos. It's nothing new, it's just got a label.
  - mikekchar 9y ago
    As a relatively old guy, I always thought that DevOps was just the new name for what people have always been doing. It is literally just now that I've realised that other people think it is different :-) Possibly younger people don't understand what system administrators used to do...
shitloadofbooks 9y ago
"DevOps Engineer" is simply just title inflation. In 2017, there's literally _no_ difference between the outcomes expected from a "Systems Engineer" and a "DevOps Engineer" - both are expected to produce HA, Automated, Well Documented Infrastructure, to allow developers to run their code. Anyone not doing that is just a _bad_ "Systems Engineer" Developers don't call themselves "Agile Developers" because they started using agile methodologies or "CI Developers" with the boom of CI, but for some reason Ops guys want to call themselves "DevOps Engineers" because they read The Phoenix Project. People talk about automation as if automation = DevOps, but I believe automation is a side-effect of the shared empathy which DevOps fosters. Because you have that empathy with your devs, you realize that their main goal is to be able to get their code in front of users _as fast as possible_. Automation is the answer to that.
- ownagefool 9y ago
  Whilst I 100% agree with you and wish it wasn't the case, title inflation is kinda necessary when the pay and authority reflects the BS titles. I'm not sure what it's like over the pond, but here in London I sit round tables with a bunch of "(security, system, application, cloud) architects" with less than a couple of years experience in the industry (or sometimes 10 years of one years experience) who think they have a remit to define how things work. In a silly bureaucracy, you have to play the game or you won't have a seat at the table. I do wish it wasn't so, but also having the title doesn't prohibit you from actually doing devops either.
- deleted 9y ago
  [deleted]
- BaronSamedi 9y ago
  "the shared empathy which DevOps fosters". I haven't heard it described like this before but I like it a lot. The way you phrased it emphasizes the "put yourself in other people's shoes" philosophy. An attitude that I would argue is the key to a successful DevOps process.
- 23david 9y ago
  > In 2017, there's literally _no_ difference between the outcomes expected from a "Systems Engineer" and a "DevOps Engineer" I don't find the title 'Systems Engineer' to be in widespread usage at tech startups in SV, so I'm not sure what to expect from one. To be honest, I would expect Systems Engineers to be working on hardware. The infrastructure automation work done at modern tech startups is best described by the title 'DevOps Engineering'. If you're posting a job where the requirements include using automation code to set up and configure CI/CD, monitoring, alerting, Databases, Analytics, AWS, GCE, Heroku or other 3rd-party SAAS tools, the job title in widespread usage that encompasses that is DevOps. The Webmaster, Ops, DBA, SRE, IT, Systems Engineer, and SysAdmin titles describe different skillsets and are therefore less useful for describing a role with these kinds of requirements. Here's an AWS-specific, but otherwise useful reference of a modern 'DevOps' skillset: https://aws.amazon.com/certification/certified-devops-engineer-professional/ https://aws.amazon.com/certification/certified-devops-engine...
robert_tweed 9y ago
I've seen two prevalent definitions of devops: 1. It's a culture where dev and ops work together (as per TFA); 2. It is the application of software engineering practises (such as version control and test automation) to ops. I prefer the second definition because it's the only one that's got anything to do with new developments in infrastructure-as-code and isn't just a new name for what competent teams were always doing anyway.
- notamy 9y ago
  Alternative third definition: Make the devs also be the ops guys
  - devoply 9y ago
    you mean the ops guys. and yeah it's just an easy way to take nerds and make them do more stuff so you can save that extra 85-100k salary. More profit for the shareholders, what's not to love.
    - provl 9y ago
      ok
    - notamy 9y ago
      I did indeed. Good catch.
  - yourapostasy 9y ago
    I've been on the sysadmin/ops end, trying to teach devs how to apply their software engineering to ops. It's not a short ramp, unfortunately; the various ops stacks now are as broad and deep as software development stacks. Worse yet, it is not Yet Another Imperative Programming Language for them to pick up. Even declarative configuration management systems embed a lot of hidden, assumed sysadmin/ops experience/skills that many devs don't arrive on the scene with, and in the vast majority of cases it was like training someone from scratch to become a sysadmin, and then start devops training. Add in cross-platform ops, and it gets more challenging. Add in enterprise demands and integration points, and it becomes a multi-year effort. More unfortunately, it is also a long ramp up time to bring sysadmins who don't know programming up to speed on software development. I'd like to know what others are doing to quickly build a devops team if they are doing that, other than "find highly-skilled senior sysadmins who already know software development", because that's my go-to approach for the time being, as it is way faster to teach someone like that Ansible if they already use Chef, for example, than pick up sysadmins or devs and cross-train them into the other field.
np422 9y ago
While I agree that DevOps needs to be a mindset/culture to be of any benefit beside ticking buzzwords checkboxes - but I really don't see any harm in giving someone a title containing the word DevOps. Someone who can do the proselytism, take initiatives to drive the adaption and at the same time be the go to guy for the technical knowledge needed for any new processes/tools. We all agree that the old waterfall inspired way of working when the developers work on their own, throws a bunch of new code down a hatch for the ops guys to deploy and they both blame the other team when something don't work is less than desirable. Maybe there are some startups that have the devops mindset already from the beginning and a few old companies that can get there on there own, but I think many need some help to get there. Perhaps they could hire someone to get them started?
jasode 9y ago
I was thinkging about that Mike Dilworth quote: "DevOps is a culture, not a role! The whole company needs to be doing DevOps for it to work." It's one of those kinds of statements that at first glance, looks really insightful and invites the head nodding up & down in agreement. However, thinking about it some more, that type of quote can be applied to anything. "Compliance is a culture not a role. Everyone from the CEO to the janitor should be aware of legal compliance. Having a compliance department is wrong." "Security is a culture not a role. Every employee should create quality passwords and follow proper encryption procedures, ..." "Marketing is a culture not a role...", "Design is a culture not a role...", "" etc, etc. Every employee should know/do everything. If you have a separate marketing team and design team, your company has failed. Yes, I can see where one ideal of devops is to "break down silos" and therefore, having a set of programmers who specialize in that is "recreating the silo" which is a contradiction. On the other hand, I can see the opposite view: if a company wholeheartedly embraces "devops", they will acknowledge that by having a team that focuses on devops. If you're a Google paying a programmer like Jeff Dean $500k+ a year, you really don't want him working on Chef/Puppet scripts instead of Tensorflow research. At a more advanced level of skills above Ansible/Docker automation is the SRE (Site Reliabity Engineers) which has a lot of overlap[1] with Devops focused engineers. A specialization in devops seems inevitable especially for larger companies that exceed 10 people. In short, I'm not sure why "devops" in particular has to rigidly adhere to Heinlein's "specialization is for insects".[2] As for other comments saying it's "title inflation", I'm confused as to the source of that sentiment. The "devops engineer" doesn't seem like a glamorous tile similar to "I'm CEO, bitch"[3]. There are no "50 Shades of Grey" movies where the female lead gets weak at the knees at the mere mention of "devops programmer". The "sanitation engineer" euphemism of "garbageman" I understand as inflation but "devops engineer" seems like a sideways or neutral label. I'm mystified by the outrage over this programming label. [1] https://en.wikipedia.org/wiki/Site_reliability_engineering#DevOps_vs_SRE https://en.wikipedia.org/wiki/Site_reliability_engineering#D... [2] https://en.wikipedia.org/wiki/Competent_man https://en.wikipedia.org/wiki/Competent_man [3] https://www.google.com/search?q="i%27m+ceo%2C+bitch" https://www.google.com/search?q="i%27m+ceo%2C+bitch"
- allsunny 9y ago
  I'll take your outrage comment has hyperbole, I've yet to see anyone that upset by it. Anyway, the frustration, I think, is it's just a new name applied to a role that always existed.
oogali 9y ago
I used to repeat this statement, but a few years of the current pattern, I disagree. (I'm going to generalize for UNIX systems, apologies in advance) Your traditional systems engineers are well versed in the ways of UNIX, but that does not automatically qualify them to wear the DevOps hat. And vice versa. Systems and DevOps are two different sides of the house. There is an entire tooling set that someone with the DevOps attribute needs to be aware of (CI/CD, tools that enable code review, etc), how they tie into the tools that the developers use, and how to troubleshoot when these things break down. You, Systems Engineer, will be called when someone can't run "npm install" and an ERRNO message scrolls down the screen. While it could be something as innocent as a missing header file from an external dependency, it's still on you to rapidly troubleshoot and resolve. You also need someone with UNIX and networking infrastructure chops. Someone who is familiar with a how to generate and read a packet capture, or maybe even use strace. You, DevOps, will be asked why does it take forever-and-a-half to run 'ls' in a directory containing 10,000 files. Scratch that, you will asked to make it faster. It's unreasonable to ask either of these folks to stretch and cover the other role, it results in a lot of resentment in how they're expected to do "two jobs" as well as complaints from developers that the DevOps/Systems Engineer's priorities and timelines don't line up with their own. Worse, you'll see abominations (a relative viewpoint) because everyone knows their tools the best. To a person with a hammer, everything looks like a nail. I've seen automated DNS failover triggered via Jenkins, and I've seen automated code deployment triggered via cron on a 5 minute schedule. DevOps is more than automation, it's about enabling developers to rapidly iterate in a sensible way that doesn't cause too much wake around a growing startup, an already functioning business, or a fast moving behemoth. It's about enabling operations to rapidly provision new applications, monitor existing applications, properly perform capacity planning, and create repeatable infrastructure ...repeatedly. And it's about both developers and operations coming together to onboard new employees into the environment in a short timeframe, so they can become productive. This is all not to say that DevOps and traditional Systems Engineers + Network Engineers should live on different teams. Ideally, they're all on the same team, with different roles, and plenty of communication and cross-training.
- seorphates 9y ago
  It looks like you're just talking about a good, old-fashioned, well-built operations team. In DevOps there is no "I can't run npm" unless the new guy is building his first personal env. In DevOps ERRNO scrolling down the screen means your kit is broken and you need to rebuild it to a known, working state, yourself (and crap like that certainly wouldn't be making it into production). In DevOps the entire organization is on board. Everyone has a role even if it's just feedback or a ring-ring fielding phone jockey and they'll know why and they'll know it's an important contribution because every lost piece of information affects delivery and quality. If you're automating, integrating, designing, orchestrating, trouble-shooting or just about any other ing-ing you can think of than you're an engineer and you can front that with whatever you like be it systems, integration, quality-assurance, software, application, automation, monitoring, cloud, reliability, hardware, database, support, etc. etc. and even devops, if you like. The difference is whether the organization you're doing all of your "engineering" for has accepted and/or is applying the DevOps "method". I've perused some "DevOps Engineer" postings and a good amount of them look like garbage idolization of this or that toolset combined with looking for someone they can beat on until their infrastructure works like they think they'd like it to. DevOps ain't infrastructure and it certainly isn't ace dev guy running the prod kit and no, it's not about "enabling" developers either. It's an operation comprising the entire cycle of delivery with expectations and requirements from all members. Given the case of some new startup and they're on the hunt and poised to ramp then what they're really looking for is a savvy engineer (or 12) that can ing-ing all. day. long. Sadly, because of business overlays and buzz-wordery and agile styles and .. certain software stacks, a more than fair share of time gets sucked right out of the actual developing and operating parts, especially if there's any palatable resistance to acceptance but, alas, it really is just another thing fronting ing.
godmodus 9y ago
Linux basement monkey/Sysadmin supporting a dev Team and soon clients, here! I dont care about nomenclature. I chose my weapons, and i can do fine with them; jenkins, artifactory, puppet, docker, vagrant and ammonite. All strung together using scala and bash scripts. Github MDs for docs and svn because it was there and the lads and ladies here like it. Anyone can choose a stack. They just have to get good at it. VALAR MORGHULIS!
ForHackernews 9y ago
In my experience, "devops" is just employer-speak for "We're too cheap to hire proper sysops or DBAs, so we'll expect you to do everything yourself."
- edgan 9y ago
  I think there is more to it than too cheap. I think it is not enough money and not enough need. If the budget is $150k and it is SV, you aren't going to hire a team. Another problem is I think Google and like companies have been vacuuming up the best of the best sysops and DBAs. This has had the effect of making them hard to find, and even if you can, you can't afford them. Then you see the conquences of companies trying to figure out an alternative plan. The results are Devops and NoOps.
dpeck 9y ago
Related, I've found that the term devops is too poisoned to be used effectively when hiring. It brings out charlatans in far greater numbers than anything else at the moment. Garden variety technician level folks who self inflate their titles and roles to try to jump from 60k to 100k overnight. It's been frustrating as they're able to get things syntactically correct for the most part that they get through early phone screening but absolutely fall apart once they reach someone with technical expertise, wasting everyone's times.
- boompowpow 9y ago
  Sounds like you need a new/retooled hiring process
- matwood 9y ago
  Devops is less about knowing the syntax of something like Terraform and more about understanding, communicating, and implementing 'devops' best practices across an organization. The early phone screen should be an easy place to weed out people. I do agree that devops is the hot new title so people are jumping on it to try and make more money.
  - dpeck 9y ago
    I mean syntax of the conversation. They're basically gaming the initial phone screenings and making us waste engineer time discovering that they don't really know anything. It's not exactly easy to get good filtering for this with first level screening but it's what we've had to do
mugsie 9y ago
Everywhere you see the word "devops" replace it with the word "empathy". That was the true meaning of DevOps - fostering empathy and communication between different groups, and to stop the siloed "throw code over the wall to Ops" mentality that existed. By having a "DevOp's Engineer" you have admitted you have failed at DevOps, because you have created a new silo. A good team will have a combination of Operations Engineers, Software Engineers, Automation Engineers and Test Engineers all of whom work together, and collaborate on tooling, processes and deployments, who also share in the rewards for a projects success, and share in the responsibility for when something goes wrong. DevOps is not a team created to "provide DevOps services to R&D" - which is the basic job description for far to many DevOps jobs postings. It is also not just handy R&D pagers, and expecting them to manage a system. Of course all of this is talking about the initial spirit of DevOps, not the current "Buy my Book", consultant driven mess that is now associated with the word.
- xorblurb 9y ago
  That's like Agile; the practices that emerged after the principles are sometimes (often?) WTF.
  - mattmanser 9y ago
    It confuses me when I read the Agile Manifesto and then look at Scrum, Lean, Kanban, XP, etc. On the one hand you have a manifesto advocating self-organising teams citing individuals over processes, while all those methodologies claim to be 'agile', but dictate your processes and how you should organise. To me, they seem incompatible. It's like several of the original signatories have developed doublethink. The one that really makes my mind boggle is that you can be a certified Scrum Master. To become the guy who tells everyone how to work? Is that not the very antithesis of the agile manifesto?
    - mugsie 9y ago
      Yeah, this gets me too. Just because someone went to a training course, and now runs a standup, and use Jira or Trello does not make what you do agile.
paki123 9y ago
At what point does testing become not worth the benefit?
dkhenry 9y ago
I agree its a culture, but there is a role to be played, and in all the cases where I have moved teams down the DevOps path it has always started with specialists who can help the team out as they transition. Then there is the operations part, no sane person would say since we do "DevOps" and "Continuous Delivery" now, so lets give ever member of the team full admin on AWS. Even in a _very_ mature organization you have snowflakes. If your DevOps team is doing its job, more and more of the workflow is being shifted all the time to the product teams and less and less is being owned by them. Then once they have delivery fully migrated, they move on to be your SRE's which is something I think most teams who have devops people will never get to.
Stranger43 9y ago
A big part of the problem is that DevOps are not really about new technologies but just a new word the same bottom up management theory that also gave rise to buzzwords like LEAN. In essence it's a shift from thinking in terms of grand strategic plans drawn up by the high management to be mindlessly implemented by the "replaceable" employees, to thinking in terms of reacting to problems as seen from the bottom of the org chart. The problems with buy in is that DevOps kills a lot of the prestige difference between different teams/silo's as it's basically about realizing that the no developer or systems architect have enough knowledge about the "real world" problem they are trying to solve to design/write anything in isolation. Or to put it differently DevOps means that the implementation and development teams owe operations a steak dinner if/when the system tumbles over and operations gets the dreaded 3am pager call, due to an untested edge case, not that the two roles merge into one.
skywhopper 9y ago
First, let me say I totally agree with the premise here--that all parts of a software engineering organization need to be on board with DevOps, buying in to the vision, learning the full stack, taking responsibility. However, this article does serve as a good example of one of the big problems with most DevOps/SRE evangelism: It's usually coming from the Dev- perspective, rather than the -Ops perspective (or better, a balanced perspective). In particular, this focuses mostly on getting code to users, and speeding up that process. While that's certainly important, it leaves out any discussion of: * security * infrastructure design * instrumentation and monitoring * planning for failure * security * performance evaluation and tuning * disaster recovery planning * security * off-hours on-call responsibilities * security Successful DevOps cultures must address these things just as much as they address code feedback loops, time-to-production, rollback strategies, code pipeline automation, etc.
- beobab 9y ago
  Speaking as someone who once had to patch up a legacy system, I appreciate items 1, 5, 8 and 10 far more than I did before.
  - ben_jones 9y ago
    Number 11 is championing 1, 5, 8, and 10, to management and other developers. Number 12 is stealing developer time for 1, 5, 8, and 10.
- dimitar 9y ago
  My experience is that even customers who are very enthusiastic about these new processes and methodologies quickly turn back to the old-school way of doing things once someone in upper management raises these bullets. And they weren't considered before because the customer Ops people were not invited to any discussions. So in order to truly get the whole organization on board you do need to include your solutions to the above points in your evangelism.. and start really early.
- David 9y ago
  Agreed, it seems like a lot of people think about devops as a way to make dev's lives better without actively harming ops. The article does have a section on getting Ops on board with devops, but it's much weaker than the previous section on getting Dev on board. Devops done right should really improve both sides, as it explicitly recognizes the tradeoffs involved with the added automation / processes / policies. I've been reading Google's Site Reliability Engineering book ([1], free online, highly recommend so far) and it lays out pretty clearly how to do devops in a more balanced way. Key points are: - Define service level indicators (SLIs), metrics you can track to see how you're doing on the things your users care about: uptime, latency, security, etc. For instance, measure uptime as successful responses / total requests. - Define service level objectives (SLOs) based on your SLIs, e.g. 99.9% successful responses / total requests (aka 99.9% uptime) - Establish an error budget which controls your deployments. Based on your SLOs you can accept a certain number of errors (failed requests, outages, slow response times, etc) for a given period. If you spend your entire error budget, no new code can be deployed. The deployment shutoff has to be agreed upon in advance and it has to be a hard and fast rule, so it's not ops' fault that dev can't deploy code. - If you reach your error budget, devs need to work on things to reduce error rate going forward instead of developing new features. It can be tests, monitoring and alerting, rollback process, or pretty much anything, but devs should know that they're doing it so they can deploy more code faster in the future. I think this makes a lot of sense and helps to align dev and ops on uptime goals. Specifically, dev's ability to push code is now directly tied to service level, and ops is not shooting for 100% reliability. Dev can choose how to spend the error budget, so ops isn't dictating how they do things; and ops knows and expects a certain number of problems, so they don't resent dev every time they push buggy code. [1] https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html
ransom1538 9y ago
Dev = "how do i start a new instance? just chmod -R 777!" Ops = "yyour code is broken!" Devops = "I am super tired."
matt_wulfeck 9y ago
Here's a thought: skilled devops engineers are indistinguishable from skilled software engineers, and vice versa. Who can be specialized anymore? Isn't devops just the compression of skill sets that we've been witnessing for the last two decades? As a developer, you now must be able to deploy, to write and maintain unit tests, and possess the knowledge to debug code deep on the system and network layer, and write code that follows up-to-date security practices. Specialization is dying.
- dkarapetyan 9y ago
  Data scientists would disagree. Specialization is alive and well.
  - matt_wulfeck 9y ago
    Only until their libraries become advanced and commoditized. Then they become indistinguishable from other software engineers.
  - kmicklas 9y ago
    I think think most skilled developers could easily do data science work, they are just more expensive to hire than data scientists.
    - dkarapetyan 9y ago
      Data science involves a lot of statistics and graduate level mathematics. I don't think skilled developers can fill the gap.
dkarapetyan 9y ago
Problem with "culture" is everyone has their own variation of it.
- sngz 9y ago
  thats the point. its the same story as when people were trying to do agile processes as a one size fit all.
jcahill84 9y ago
Yes, yes, a thousand times yes.
djohnston 9y ago
My official title is DevOps, but my coworkers know that I'm just the most cloud-oriented developer on the team. In that regard I also regularly push code into our products, with an insistence that I am not simply the guy that does AWS/Jenkins/Ansible work. So, in one respect I entirely agree that DevOps is just another developer, at least the way I practice it. That being said, it's naive to think that DevOps isn't a specialization, or that everyone writing your code is going to be equally knowledgable on a product landscape as complex as AWS. Ultimately where you fall on the spectrum of developer and ops is company specific, even project specific, and to try and make some generalizable dogma on the issue is a waste of time.
slackingoff2017 9y ago
Devops just means your IT people are software engineers. That's it. Web systems are so complex (and automated) to manage these days that you need to be able to write code to manage the infastructure. Google has been doing this for ages without calling it Devops. As far as I'm aware all of their main support people are software devs
- praneshp 9y ago
  > Google has been doing this for ages without calling it Devops Are you of the opinion that DevOps, like Agile, is a buzzword and money making opportunity? I had a brief stint in a DevOps-ish role and that was my conclusion.
  - slackingoff2017 9y ago
    100% from the two companies I've seen implementing it. All it means in practice is "fuck, our IT guys don't understand this docker thing. We need a software guy to write up stuff to automate all this" I looked at a couple Devops position and saw the same. Basically "we need a software engineer with networking skills". IT jobs are quickly vanishing. All the stuff that doesn't require software development has been automated away besides things anyone can do like wire some machines to a switch. Everything is either really easy or quite hard and it's hard to hire a guy that knows software at an IT salary. Now we have Devops, so that execs can justify hiring software engineers to maintain the network.
m23khan 9y ago
I am a DevOps Engineer by trade and title. Here is my take on this: It is a role - not something that can be done by OPS or DEV or SEC or what have you. Allow me to explain my 2 cents worth: - With the amounts of tooling (CI/CD pipeline, infrastructure monitoring, infra. provisioning, config. management, etc.), you do not want your DEVs or your full-time admins focusing on this bit at-all. It is good if they have some know-how (or even if they do the initial groundwork) but if your DEVs or sys admins are focusing on what 'entails' as DevOps -- then Houston, we got a problem. Because this means that DEV or sys admin or what have you is not focusing 100% on their role. - Why is 'DevOps' getting so much flak?? How about titles/roles such as 'scrum master' or 'Agile Practitioner' or 'Product Owner'? - I am not a noob by any means (~10 yr IT experience) and I am past the stage of getting excited by a job title. However, the 'DevOps' title given to a 'DevOps' guy is a correct move. AND, believe-it-or-not, it is a IT specialization -- just like 'cloud' is. Just because everything is a code doesn't mean you don't need a dedicated person who will not only preach DevOps -- but will also 'implement' DevOps -- and that fella should be called 'DevOps' -- just like 'Agile Coach' ;-)
- ErrantX 9y ago
  I'd have a contrary view which is that yes you DO want your engineers to understand dev and ops (and sec and change etc etc) and actively be able to own a service through its lifecycle. Those engineers might have different focuses; so maybe building a data store, or an api, or a Jenkins system, or an artefact store.... If you leave your cicd tooling and toolchains to a team then you create a dependency, and for me a key part of devops is reducing dependencies. The suggestion in your post is that by blurring the traditional roles you make those people less productive. And therefore the solution is to create a new role. That's fine: but that's the old thinking (IMO) hanging around. The original intent of devops would definitely run counter to that :)
- lmickh 9y ago
  Would argue that you have a tools/service team. The fact that you failed to mention changing what devs and sysadmins consider when "focusing 100%" on their role indicates that what you are talking about isn't the concept of DevOps in the article. Saying you have a DevOps team is like saying you have Lean team, a Theory of Constraints team, or a Just-in-time manufacturing team. DevOps doesn't say that devs shouldn't write code any more then Theory of Constraints says a guy that paints car hoods should be building brake pads. It is about the fact that they should work together to ensure the organization as a whole meets the goals. Not as isolated units that only prioritize their own goals. I've seen the same flak about Agile and scrum roles for the same reason. People think they can buy the DevOps/Agile/whatever. Only difference with DevOps is rest is that it is getting more noise.
bsharitt 9y ago
I've been browsing around job boards lately and see a lot of "DevOps Engineer" positions, but the descriptions of each vary widely. Some of them appear to be "real" DevOps roles, that is they sit somewhere between the traditional dev and ops roles and the basic mission is to get code out to users fast. But then many of them are simply rebranded systems engineers, while others are dev roles where managing infrastructure is part of the job, while other job description have read like very basic sysadmin job descriptions.
jnevelson 9y ago
Here's my point of view as someone who's title currently is "Senior DevOps Engineer", and used to be a software engineer for many years: DevOps has multiple meanings - in some (usually smaller) companies it can be thought of as a philosophy in that all developers are responsible for managing production infrastructure and their deployments. As a company scales, and the codebase grows, people have to start specializing. It's just not feasible for everyone to have perfectly overlapping skillsets and be completely replaceable by each other. It happens with frontend/backend, and it happens with "DevOps". I focus on our production infrastructure, making sure deployments are smooth, our database has backups, etc. That doesn't mean that other developers are completely detached from our production environment - if there's an issue in production multiple parties are involved in working through it and fixing. I personally interchange the title DevOps Engineer with Site Reliability Engineer, Infrastructure Engineer, Ops/SysOps Engineer, etc. Different companies have different titles, but I'd wager that the majority of those positions tend to perform work of a similar nature. Depending on the size of the company, some may specialize even more (i.e. a large company may have SRE to focus on production/monitoring, and someone else to focus on deployments). It definitely varies company to company, but by and large I think it can all be grouped under "software operations", which itself is a subset of software engineering.

236 points by mromnia 9y ago

Yes yes, we get it, it's a mindset not a role. Now could we hire a DevOps guy to setup that CI/CD pipeline, ChatOps stuff, Kubernetes and dev/staging/prod flow so the coders can ship. ;) The original thought was beautiful but the way I see it: DevOps is a branch where you specialise. After school people might learn front/back/db stuff and then specialise to "DevOps stuff" -- or then you have more sysops kind of people who start to venture in the land of automating their work. Both are needed imho.

enraged_camel 9y ago
>>ChatOps stuff That is so 2013! I think you meant SlackOps. ;)
nraynaud 9y ago
I hate specialization, and I want the person who develop something to think about how that stuff will be deployed and kept online. In particular, some tradeoffs have to be made between the development side and the production side, and I want the same person to see both side and decide with a technical criterion (or just choose one option, and later reverse course and change it), not on a "it's not my problem" criterion.
- preordained 9y ago
  There is too much. It's specialization out of necessity. I know enough "DevOps" to be dangerous, but in addition to having a deep knowledge of our product's codebase, the multiple features of our product, configurations, etc. there are not enough hours in the day to master (not simply be familiar with) Jenkins pipeline, Docker, AWS, Ansible, Zookeeper, etc, etc. 99% of the people who could claim to have knowledge of all the above in their organization either have a very shallow (read: not useful) knowledge, OR their product itself is simply the gluing together of various DevOps technologies listed, so there is little more to know outside that.
  - cat199 9y ago
    it is definitely possible to have fairly deep level of knowledge at all layers it just takes effort. That said, the larger the project, the more the need for specialization and the less possible it is for someone to keep up with all changes and be a domain expert.
    - aNoob7000 9y ago
      I disagree. Maybe, there's a couple of guys/gals around that can go up and down the software and hardware stack and know where issues may lie when there's a problem, but I doubt there are many. As a someone that's been in the tech industry a very long time, I came to the realization a long time ago that I couldn't be the expert at everything. Can I dabble in Oracle, Windows, Linux? Sure, would I want to bet my job on how much I know some of these products in the time of crisis, hell No!!!
      - nraynaud 9y ago
        My last job as an employee, I had to install AWS, develop some server software, develop some web facing software, develop some embedded software (with a bit of easy real time), get the freaking 2G to work, get the sensors to work properly, model the mechanical parts to install the embedded device on site, order all that crap (electronics and mechanics) from the providers, create the boxes, try to think about lightnings and how not to fry the customer's equipment. And get the sandcastle to stay put. That's what happens when you work at startups, there is no plethora of employees, and a lot of work.

ciguy 9y ago

As a DevOps consultant, I think it can actually be both. Culture is central to DevOps but someone has to lead. Sometimes this can be the CTO but often it helps to have a dedicated advocate, especially within larger orgs/teams. Essentially it boils down to semantics. Often DevOps engineers are one part evangelist and one part automation engineer. The software industry has just chosen to call those people "DevOps".

holydude 9y ago

I do not know. We always had "DevOPS" aka sysadmin that knew C,perl and bunch of others. Then it became too much for a single person to know everything and we invented "more ops" and "more programmer". Now we have better tools albeit more complex and we have decided that we again can have this "DevOPS" person whatever that means. But again we are facing this dilemma when there are literally thousands of different tools and ecosystems and you just simply cannot be an EXPERT or even ADVANCED in most of them. We should clearly define what we expect from the role and the candidate. We can't waste other people's time by hiring for DevOPS when we clearly want a PowerShell programmer,windows administrator and database admin in a single package

dozzie 9y ago
> But again we are facing this dilemma when there are literally thousands of different tools and ecosystems and you just simply cannot be an EXPERT or even ADVANCED in most of them. Though there are only a handful of ideas and principles, and all of these thousands of different tools are just reimplementations of some of them, and usually not too fancy at that.
- holydude 9y ago
  That's correct but you can't learn all the nuances and specifics of every tool there is.
  - dozzie 9y ago
    Of course you don't, but those nuances and specifics are usually superficial and don't change how you approach your environment. You think there is that much of a difference between running Puppet, CFEngine, or Chef?
    - holydude 9y ago
      They are not superficial and they definitely change how you approach your environment. I have worked extensively with both CFEngine and Puppet and I can assure you that they have enough differences especially when you consider large heterogenous environments that in no way you can consider using one as a substitute for the other.

benjaminwootton 9y ago

Having worked on over 100 DevOps initiatives, I'm inclined to disagree. 8 years into DevOps, automation, cloud computing etc, it's time to accept that DevOps is a specialism and a function that companies - especially big ones - need to put into place in order to adopt the principles. See here for further thoughts - https://devops.com/im-happy-devops-engineer-job-title/ https://devops.com/im-happy-devops-engineer-job-title/

signal11 9y ago
I don't really care about titles so if calling someone a DevOps Engineer helps them and connects the right engineers to the right roles, great. Fighting against this use of "DevOps" would be like fighting against how the word "literally" is misused to mean "figuratively". However for me, "DevOps" is more than just CI/CD and tech automation, it's about not working in silos and having empathy for others in the team (and the customer!). This is a culture and mindset that everyone in the team needs to have.
inopinatus 9y ago
I disagree with your disagreement. There's nothing we do under the umbrella of "DevOps" that wasn't part of competent service ownership twenty years ago. I read your article and the point that stood out for me was the notion that the skills to drive tools like chef, puppet are somehow new and different. But I know better, because although I use those tools today, back in the day we used cfengine and a bunch of declaratively/injectively configured shell & perl scripts to achieve basically exactly the same effect. Back then we called this "systems administration". But we were most definitely working with developers and running the SDLC across silos. It's nothing new, it's just got a label.
- mikekchar 9y ago
  As a relatively old guy, I always thought that DevOps was just the new name for what people have always been doing. It is literally just now that I've realised that other people think it is different :-) Possibly younger people don't understand what system administrators used to do...

shitloadofbooks 9y ago

"DevOps Engineer" is simply just title inflation. In 2017, there's literally _no_ difference between the outcomes expected from a "Systems Engineer" and a "DevOps Engineer" - both are expected to produce HA, Automated, Well Documented Infrastructure, to allow developers to run their code. Anyone not doing that is just a _bad_ "Systems Engineer" Developers don't call themselves "Agile Developers" because they started using agile methodologies or "CI Developers" with the boom of CI, but for some reason Ops guys want to call themselves "DevOps Engineers" because they read The Phoenix Project. People talk about automation as if automation = DevOps, but I believe automation is a side-effect of the shared empathy which DevOps fosters. Because you have that empathy with your devs, you realize that their main goal is to be able to get their code in front of users _as fast as possible_. Automation is the answer to that.

ownagefool 9y ago
Whilst I 100% agree with you and wish it wasn't the case, title inflation is kinda necessary when the pay and authority reflects the BS titles. I'm not sure what it's like over the pond, but here in London I sit round tables with a bunch of "(security, system, application, cloud) architects" with less than a couple of years experience in the industry (or sometimes 10 years of one years experience) who think they have a remit to define how things work. In a silly bureaucracy, you have to play the game or you won't have a seat at the table. I do wish it wasn't so, but also having the title doesn't prohibit you from actually doing devops either.
deleted 9y ago
[deleted]
BaronSamedi 9y ago
"the shared empathy which DevOps fosters". I haven't heard it described like this before but I like it a lot. The way you phrased it emphasizes the "put yourself in other people's shoes" philosophy. An attitude that I would argue is the key to a successful DevOps process.
23david 9y ago
> In 2017, there's literally _no_ difference between the outcomes expected from a "Systems Engineer" and a "DevOps Engineer" I don't find the title 'Systems Engineer' to be in widespread usage at tech startups in SV, so I'm not sure what to expect from one. To be honest, I would expect Systems Engineers to be working on hardware. The infrastructure automation work done at modern tech startups is best described by the title 'DevOps Engineering'. If you're posting a job where the requirements include using automation code to set up and configure CI/CD, monitoring, alerting, Databases, Analytics, AWS, GCE, Heroku or other 3rd-party SAAS tools, the job title in widespread usage that encompasses that is DevOps. The Webmaster, Ops, DBA, SRE, IT, Systems Engineer, and SysAdmin titles describe different skillsets and are therefore less useful for describing a role with these kinds of requirements. Here's an AWS-specific, but otherwise useful reference of a modern 'DevOps' skillset: https://aws.amazon.com/certification/certified-devops-engineer-professional/ https://aws.amazon.com/certification/certified-devops-engine...

robert_tweed 9y ago

I've seen two prevalent definitions of devops: 1. It's a culture where dev and ops work together (as per TFA); 2. It is the application of software engineering practises (such as version control and test automation) to ops. I prefer the second definition because it's the only one that's got anything to do with new developments in infrastructure-as-code and isn't just a new name for what competent teams were always doing anyway.

notamy 9y ago
Alternative third definition: Make the devs also be the ops guys
- devoply 9y ago
  you mean the ops guys. and yeah it's just an easy way to take nerds and make them do more stuff so you can save that extra 85-100k salary. More profit for the shareholders, what's not to love.
  - provl 9y ago
    ok
  - notamy 9y ago
    I did indeed. Good catch.
- yourapostasy 9y ago
  I've been on the sysadmin/ops end, trying to teach devs how to apply their software engineering to ops. It's not a short ramp, unfortunately; the various ops stacks now are as broad and deep as software development stacks. Worse yet, it is not Yet Another Imperative Programming Language for them to pick up. Even declarative configuration management systems embed a lot of hidden, assumed sysadmin/ops experience/skills that many devs don't arrive on the scene with, and in the vast majority of cases it was like training someone from scratch to become a sysadmin, and then start devops training. Add in cross-platform ops, and it gets more challenging. Add in enterprise demands and integration points, and it becomes a multi-year effort. More unfortunately, it is also a long ramp up time to bring sysadmins who don't know programming up to speed on software development. I'd like to know what others are doing to quickly build a devops team if they are doing that, other than "find highly-skilled senior sysadmins who already know software development", because that's my go-to approach for the time being, as it is way faster to teach someone like that Ansible if they already use Chef, for example, than pick up sysadmins or devs and cross-train them into the other field.

np422 9y ago

While I agree that DevOps needs to be a mindset/culture to be of any benefit beside ticking buzzwords checkboxes - but I really don't see any harm in giving someone a title containing the word DevOps. Someone who can do the proselytism, take initiatives to drive the adaption and at the same time be the go to guy for the technical knowledge needed for any new processes/tools. We all agree that the old waterfall inspired way of working when the developers work on their own, throws a bunch of new code down a hatch for the ops guys to deploy and they both blame the other team when something don't work is less than desirable. Maybe there are some startups that have the devops mindset already from the beginning and a few old companies that can get there on there own, but I think many need some help to get there. Perhaps they could hire someone to get them started?

jasode 9y ago

I was thinkging about that Mike Dilworth quote: "DevOps is a culture, not a role! The whole company needs to be doing DevOps for it to work." It's one of those kinds of statements that at first glance, looks really insightful and invites the head nodding up & down in agreement. However, thinking about it some more, that type of quote can be applied to anything. "Compliance is a culture not a role. Everyone from the CEO to the janitor should be aware of legal compliance. Having a compliance department is wrong." "Security is a culture not a role. Every employee should create quality passwords and follow proper encryption procedures, ..." "Marketing is a culture not a role...", "Design is a culture not a role...", "" etc, etc. Every employee should know/do everything. If you have a separate marketing team and design team, your company has failed. Yes, I can see where one ideal of devops is to "break down silos" and therefore, having a set of programmers who specialize in that is "recreating the silo" which is a contradiction. On the other hand, I can see the opposite view: if a company wholeheartedly embraces "devops", they will acknowledge that by having a team that focuses on devops. If you're a Google paying a programmer like Jeff Dean $500k+ a year, you really don't want him working on Chef/Puppet scripts instead of Tensorflow research. At a more advanced level of skills above Ansible/Docker automation is the SRE (Site Reliabity Engineers) which has a lot of overlap[1] with Devops focused engineers. A specialization in devops seems inevitable especially for larger companies that exceed 10 people. In short, I'm not sure why "devops" in particular has to rigidly adhere to Heinlein's "specialization is for insects".[2] As for other comments saying it's "title inflation", I'm confused as to the source of that sentiment. The "devops engineer" doesn't seem like a glamorous tile similar to "I'm CEO, bitch"[3]. There are no "50 Shades of Grey" movies where the female lead gets weak at the knees at the mere mention of "devops programmer". The "sanitation engineer" euphemism of "garbageman" I understand as inflation but "devops engineer" seems like a sideways or neutral label. I'm mystified by the outrage over this programming label. [1] https://en.wikipedia.org/wiki/Site_reliability_engineering#DevOps_vs_SRE https://en.wikipedia.org/wiki/Site_reliability_engineering#D... [2] https://en.wikipedia.org/wiki/Competent_man https://en.wikipedia.org/wiki/Competent_man [3] https://www.google.com/search?q="i%27m+ceo%2C+bitch" https://www.google.com/search?q="i%27m+ceo%2C+bitch"

allsunny 9y ago
I'll take your outrage comment has hyperbole, I've yet to see anyone that upset by it. Anyway, the frustration, I think, is it's just a new name applied to a role that always existed.

oogali 9y ago

I used to repeat this statement, but a few years of the current pattern, I disagree. (I'm going to generalize for UNIX systems, apologies in advance) Your traditional systems engineers are well versed in the ways of UNIX, but that does not automatically qualify them to wear the DevOps hat. And vice versa. Systems and DevOps are two different sides of the house. There is an entire tooling set that someone with the DevOps attribute needs to be aware of (CI/CD, tools that enable code review, etc), how they tie into the tools that the developers use, and how to troubleshoot when these things break down. You, Systems Engineer, will be called when someone can't run "npm install" and an ERRNO message scrolls down the screen. While it could be something as innocent as a missing header file from an external dependency, it's still on you to rapidly troubleshoot and resolve. You also need someone with UNIX and networking infrastructure chops. Someone who is familiar with a how to generate and read a packet capture, or maybe even use strace. You, DevOps, will be asked why does it take forever-and-a-half to run 'ls' in a directory containing 10,000 files. Scratch that, you will asked to make it faster. It's unreasonable to ask either of these folks to stretch and cover the other role, it results in a lot of resentment in how they're expected to do "two jobs" as well as complaints from developers that the DevOps/Systems Engineer's priorities and timelines don't line up with their own. Worse, you'll see abominations (a relative viewpoint) because everyone knows their tools the best. To a person with a hammer, everything looks like a nail. I've seen automated DNS failover triggered via Jenkins, and I've seen automated code deployment triggered via cron on a 5 minute schedule. DevOps is more than automation, it's about enabling developers to rapidly iterate in a sensible way that doesn't cause too much wake around a growing startup, an already functioning business, or a fast moving behemoth. It's about enabling operations to rapidly provision new applications, monitor existing applications, properly perform capacity planning, and create repeatable infrastructure ...repeatedly. And it's about both developers and operations coming together to onboard new employees into the environment in a short timeframe, so they can become productive. This is all not to say that DevOps and traditional Systems Engineers + Network Engineers should live on different teams. Ideally, they're all on the same team, with different roles, and plenty of communication and cross-training.

seorphates 9y ago
It looks like you're just talking about a good, old-fashioned, well-built operations team. In DevOps there is no "I can't run npm" unless the new guy is building his first personal env. In DevOps ERRNO scrolling down the screen means your kit is broken and you need to rebuild it to a known, working state, yourself (and crap like that certainly wouldn't be making it into production). In DevOps the entire organization is on board. Everyone has a role even if it's just feedback or a ring-ring fielding phone jockey and they'll know why and they'll know it's an important contribution because every lost piece of information affects delivery and quality. If you're automating, integrating, designing, orchestrating, trouble-shooting or just about any other ing-ing you can think of than you're an engineer and you can front that with whatever you like be it systems, integration, quality-assurance, software, application, automation, monitoring, cloud, reliability, hardware, database, support, etc. etc. and even devops, if you like. The difference is whether the organization you're doing all of your "engineering" for has accepted and/or is applying the DevOps "method". I've perused some "DevOps Engineer" postings and a good amount of them look like garbage idolization of this or that toolset combined with looking for someone they can beat on until their infrastructure works like they think they'd like it to. DevOps ain't infrastructure and it certainly isn't ace dev guy running the prod kit and no, it's not about "enabling" developers either. It's an operation comprising the entire cycle of delivery with expectations and requirements from all members. Given the case of some new startup and they're on the hunt and poised to ramp then what they're really looking for is a savvy engineer (or 12) that can ing-ing all. day. long. Sadly, because of business overlays and buzz-wordery and agile styles and .. certain software stacks, a more than fair share of time gets sucked right out of the actual developing and operating parts, especially if there's any palatable resistance to acceptance but, alas, it really is just another thing fronting ing.

godmodus 9y ago

Linux basement monkey/Sysadmin supporting a dev Team and soon clients, here! I dont care about nomenclature. I chose my weapons, and i can do fine with them; jenkins, artifactory, puppet, docker, vagrant and ammonite. All strung together using scala and bash scripts. Github MDs for docs and svn because it was there and the lads and ladies here like it. Anyone can choose a stack. They just have to get good at it. VALAR MORGHULIS!

ForHackernews 9y ago

In my experience, "devops" is just employer-speak for "We're too cheap to hire proper sysops or DBAs, so we'll expect you to do everything yourself."

edgan 9y ago
I think there is more to it than too cheap. I think it is not enough money and not enough need. If the budget is $150k and it is SV, you aren't going to hire a team. Another problem is I think Google and like companies have been vacuuming up the best of the best sysops and DBAs. This has had the effect of making them hard to find, and even if you can, you can't afford them. Then you see the conquences of companies trying to figure out an alternative plan. The results are Devops and NoOps.

dpeck 9y ago

Related, I've found that the term devops is too poisoned to be used effectively when hiring. It brings out charlatans in far greater numbers than anything else at the moment. Garden variety technician level folks who self inflate their titles and roles to try to jump from 60k to 100k overnight. It's been frustrating as they're able to get things syntactically correct for the most part that they get through early phone screening but absolutely fall apart once they reach someone with technical expertise, wasting everyone's times.

boompowpow 9y ago
Sounds like you need a new/retooled hiring process
matwood 9y ago
Devops is less about knowing the syntax of something like Terraform and more about understanding, communicating, and implementing 'devops' best practices across an organization. The early phone screen should be an easy place to weed out people. I do agree that devops is the hot new title so people are jumping on it to try and make more money.
- dpeck 9y ago
  I mean syntax of the conversation. They're basically gaming the initial phone screenings and making us waste engineer time discovering that they don't really know anything. It's not exactly easy to get good filtering for this with first level screening but it's what we've had to do

mugsie 9y ago

Everywhere you see the word "devops" replace it with the word "empathy". That was the true meaning of DevOps - fostering empathy and communication between different groups, and to stop the siloed "throw code over the wall to Ops" mentality that existed. By having a "DevOp's Engineer" you have admitted you have failed at DevOps, because you have created a new silo. A good team will have a combination of Operations Engineers, Software Engineers, Automation Engineers and Test Engineers all of whom work together, and collaborate on tooling, processes and deployments, who also share in the rewards for a projects success, and share in the responsibility for when something goes wrong. DevOps is not a team created to "provide DevOps services to R&D" - which is the basic job description for far to many DevOps jobs postings. It is also not just handy R&D pagers, and expecting them to manage a system. Of course all of this is talking about the initial spirit of DevOps, not the current "Buy my Book", consultant driven mess that is now associated with the word.

xorblurb 9y ago
That's like Agile; the practices that emerged after the principles are sometimes (often?) WTF.
- mattmanser 9y ago
  It confuses me when I read the Agile Manifesto and then look at Scrum, Lean, Kanban, XP, etc. On the one hand you have a manifesto advocating self-organising teams citing individuals over processes, while all those methodologies claim to be 'agile', but dictate your processes and how you should organise. To me, they seem incompatible. It's like several of the original signatories have developed doublethink. The one that really makes my mind boggle is that you can be a certified Scrum Master. To become the guy who tells everyone how to work? Is that not the very antithesis of the agile manifesto?
  - mugsie 9y ago
    Yeah, this gets me too. Just because someone went to a training course, and now runs a standup, and use Jira or Trello does not make what you do agile.

paki123 9y ago

At what point does testing become not worth the benefit?

dkhenry 9y ago

I agree its a culture, but there is a role to be played, and in all the cases where I have moved teams down the DevOps path it has always started with specialists who can help the team out as they transition. Then there is the operations part, no sane person would say since we do "DevOps" and "Continuous Delivery" now, so lets give ever member of the team full admin on AWS. Even in a _very_ mature organization you have snowflakes. If your DevOps team is doing its job, more and more of the workflow is being shifted all the time to the product teams and less and less is being owned by them. Then once they have delivery fully migrated, they move on to be your SRE's which is something I think most teams who have devops people will never get to.

Stranger43 9y ago

A big part of the problem is that DevOps are not really about new technologies but just a new word the same bottom up management theory that also gave rise to buzzwords like LEAN. In essence it's a shift from thinking in terms of grand strategic plans drawn up by the high management to be mindlessly implemented by the "replaceable" employees, to thinking in terms of reacting to problems as seen from the bottom of the org chart. The problems with buy in is that DevOps kills a lot of the prestige difference between different teams/silo's as it's basically about realizing that the no developer or systems architect have enough knowledge about the "real world" problem they are trying to solve to design/write anything in isolation. Or to put it differently DevOps means that the implementation and development teams owe operations a steak dinner if/when the system tumbles over and operations gets the dreaded 3am pager call, due to an untested edge case, not that the two roles merge into one.

skywhopper 9y ago

First, let me say I totally agree with the premise here--that all parts of a software engineering organization need to be on board with DevOps, buying in to the vision, learning the full stack, taking responsibility. However, this article does serve as a good example of one of the big problems with most DevOps/SRE evangelism: It's usually coming from the Dev- perspective, rather than the -Ops perspective (or better, a balanced perspective). In particular, this focuses mostly on getting code to users, and speeding up that process. While that's certainly important, it leaves out any discussion of: * security * infrastructure design * instrumentation and monitoring * planning for failure * security * performance evaluation and tuning * disaster recovery planning * security * off-hours on-call responsibilities * security Successful DevOps cultures must address these things just as much as they address code feedback loops, time-to-production, rollback strategies, code pipeline automation, etc.

beobab 9y ago
Speaking as someone who once had to patch up a legacy system, I appreciate items 1, 5, 8 and 10 far more than I did before.
- ben_jones 9y ago
  Number 11 is championing 1, 5, 8, and 10, to management and other developers. Number 12 is stealing developer time for 1, 5, 8, and 10.
dimitar 9y ago
My experience is that even customers who are very enthusiastic about these new processes and methodologies quickly turn back to the old-school way of doing things once someone in upper management raises these bullets. And they weren't considered before because the customer Ops people were not invited to any discussions. So in order to truly get the whole organization on board you do need to include your solutions to the above points in your evangelism.. and start really early.
David 9y ago
Agreed, it seems like a lot of people think about devops as a way to make dev's lives better without actively harming ops. The article does have a section on getting Ops on board with devops, but it's much weaker than the previous section on getting Dev on board. Devops done right should really improve both sides, as it explicitly recognizes the tradeoffs involved with the added automation / processes / policies. I've been reading Google's Site Reliability Engineering book ([1], free online, highly recommend so far) and it lays out pretty clearly how to do devops in a more balanced way. Key points are: - Define service level indicators (SLIs), metrics you can track to see how you're doing on the things your users care about: uptime, latency, security, etc. For instance, measure uptime as successful responses / total requests. - Define service level objectives (SLOs) based on your SLIs, e.g. 99.9% successful responses / total requests (aka 99.9% uptime) - Establish an error budget which controls your deployments. Based on your SLOs you can accept a certain number of errors (failed requests, outages, slow response times, etc) for a given period. If you spend your entire error budget, no new code can be deployed. The deployment shutoff has to be agreed upon in advance and it has to be a hard and fast rule, so it's not ops' fault that dev can't deploy code. - If you reach your error budget, devs need to work on things to reduce error rate going forward instead of developing new features. It can be tests, monitoring and alerting, rollback process, or pretty much anything, but devs should know that they're doing it so they can deploy more code faster in the future. I think this makes a lot of sense and helps to align dev and ops on uptime goals. Specifically, dev's ability to push code is now directly tied to service level, and ops is not shooting for 100% reliability. Dev can choose how to spend the error budget, so ops isn't dictating how they do things; and ops knows and expects a certain number of problems, so they don't resent dev every time they push buggy code. [1] https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html

ransom1538 9y ago

Dev = "how do i start a new instance? just chmod -R 777!" Ops = "yyour code is broken!" Devops = "I am super tired."

matt_wulfeck 9y ago

Here's a thought: skilled devops engineers are indistinguishable from skilled software engineers, and vice versa. Who can be specialized anymore? Isn't devops just the compression of skill sets that we've been witnessing for the last two decades? As a developer, you now must be able to deploy, to write and maintain unit tests, and possess the knowledge to debug code deep on the system and network layer, and write code that follows up-to-date security practices. Specialization is dying.

dkarapetyan 9y ago
Data scientists would disagree. Specialization is alive and well.
- matt_wulfeck 9y ago
  Only until their libraries become advanced and commoditized. Then they become indistinguishable from other software engineers.
- kmicklas 9y ago
  I think think most skilled developers could easily do data science work, they are just more expensive to hire than data scientists.
  - dkarapetyan 9y ago
    Data science involves a lot of statistics and graduate level mathematics. I don't think skilled developers can fill the gap.

dkarapetyan 9y ago

Problem with "culture" is everyone has their own variation of it.

sngz 9y ago
thats the point. its the same story as when people were trying to do agile processes as a one size fit all.

jcahill84 9y ago

Yes, yes, a thousand times yes.

djohnston 9y ago

My official title is DevOps, but my coworkers know that I'm just the most cloud-oriented developer on the team. In that regard I also regularly push code into our products, with an insistence that I am not simply the guy that does AWS/Jenkins/Ansible work. So, in one respect I entirely agree that DevOps is just another developer, at least the way I practice it. That being said, it's naive to think that DevOps isn't a specialization, or that everyone writing your code is going to be equally knowledgable on a product landscape as complex as AWS. Ultimately where you fall on the spectrum of developer and ops is company specific, even project specific, and to try and make some generalizable dogma on the issue is a waste of time.

slackingoff2017 9y ago

Devops just means your IT people are software engineers. That's it. Web systems are so complex (and automated) to manage these days that you need to be able to write code to manage the infastructure. Google has been doing this for ages without calling it Devops. As far as I'm aware all of their main support people are software devs

praneshp 9y ago
> Google has been doing this for ages without calling it Devops Are you of the opinion that DevOps, like Agile, is a buzzword and money making opportunity? I had a brief stint in a DevOps-ish role and that was my conclusion.
- slackingoff2017 9y ago
  100% from the two companies I've seen implementing it. All it means in practice is "fuck, our IT guys don't understand this docker thing. We need a software guy to write up stuff to automate all this" I looked at a couple Devops position and saw the same. Basically "we need a software engineer with networking skills". IT jobs are quickly vanishing. All the stuff that doesn't require software development has been automated away besides things anyone can do like wire some machines to a switch. Everything is either really easy or quite hard and it's hard to hire a guy that knows software at an IT salary. Now we have Devops, so that execs can justify hiring software engineers to maintain the network.

m23khan 9y ago

I am a DevOps Engineer by trade and title. Here is my take on this: It is a role - not something that can be done by OPS or DEV or SEC or what have you. Allow me to explain my 2 cents worth: - With the amounts of tooling (CI/CD pipeline, infrastructure monitoring, infra. provisioning, config. management, etc.), you do not want your DEVs or your full-time admins focusing on this bit at-all. It is good if they have some know-how (or even if they do the initial groundwork) but if your DEVs or sys admins are focusing on what 'entails' as DevOps -- then Houston, we got a problem. Because this means that DEV or sys admin or what have you is not focusing 100% on their role. - Why is 'DevOps' getting so much flak?? How about titles/roles such as 'scrum master' or 'Agile Practitioner' or 'Product Owner'? - I am not a noob by any means (~10 yr IT experience) and I am past the stage of getting excited by a job title. However, the 'DevOps' title given to a 'DevOps' guy is a correct move. AND, believe-it-or-not, it is a IT specialization -- just like 'cloud' is. Just because everything is a code doesn't mean you don't need a dedicated person who will not only preach DevOps -- but will also 'implement' DevOps -- and that fella should be called 'DevOps' -- just like 'Agile Coach' ;-)

ErrantX 9y ago
I'd have a contrary view which is that yes you DO want your engineers to understand dev and ops (and sec and change etc etc) and actively be able to own a service through its lifecycle. Those engineers might have different focuses; so maybe building a data store, or an api, or a Jenkins system, or an artefact store.... If you leave your cicd tooling and toolchains to a team then you create a dependency, and for me a key part of devops is reducing dependencies. The suggestion in your post is that by blurring the traditional roles you make those people less productive. And therefore the solution is to create a new role. That's fine: but that's the old thinking (IMO) hanging around. The original intent of devops would definitely run counter to that :)
lmickh 9y ago
Would argue that you have a tools/service team. The fact that you failed to mention changing what devs and sysadmins consider when "focusing 100%" on their role indicates that what you are talking about isn't the concept of DevOps in the article. Saying you have a DevOps team is like saying you have Lean team, a Theory of Constraints team, or a Just-in-time manufacturing team. DevOps doesn't say that devs shouldn't write code any more then Theory of Constraints says a guy that paints car hoods should be building brake pads. It is about the fact that they should work together to ensure the organization as a whole meets the goals. Not as isolated units that only prioritize their own goals. I've seen the same flak about Agile and scrum roles for the same reason. People think they can buy the DevOps/Agile/whatever. Only difference with DevOps is rest is that it is getting more noise.

bsharitt 9y ago

I've been browsing around job boards lately and see a lot of "DevOps Engineer" positions, but the descriptions of each vary widely. Some of them appear to be "real" DevOps roles, that is they sit somewhere between the traditional dev and ops roles and the basic mission is to get code out to users fast. But then many of them are simply rebranded systems engineers, while others are dev roles where managing infrastructure is part of the job, while other job description have read like very basic sysadmin job descriptions.

jnevelson 9y ago

Here's my point of view as someone who's title currently is "Senior DevOps Engineer", and used to be a software engineer for many years: DevOps has multiple meanings - in some (usually smaller) companies it can be thought of as a philosophy in that all developers are responsible for managing production infrastructure and their deployments. As a company scales, and the codebase grows, people have to start specializing. It's just not feasible for everyone to have perfectly overlapping skillsets and be completely replaceable by each other. It happens with frontend/backend, and it happens with "DevOps". I focus on our production infrastructure, making sure deployments are smooth, our database has backups, etc. That doesn't mean that other developers are completely detached from our production environment - if there's an issue in production multiple parties are involved in working through it and fixing. I personally interchange the title DevOps Engineer with Site Reliability Engineer, Infrastructure Engineer, Ops/SysOps Engineer, etc. Different companies have different titles, but I'd wager that the majority of those positions tend to perform work of a similar nature. Depending on the size of the company, some may specialize even more (i.e. a large company may have SRE to focus on production/monitoring, and someone else to focus on deployments). It definitely varies company to company, but by and large I think it can all be grouped under "software operations", which itself is a subset of software engineering.