26 ms·

DevOps is a failure

• wizofaus 4y ago
  I just quit a job partly because we lost our key DevOps guy and no serious effort was made to replace them. As a result I ended up wasting huge amounts of my time dealing with operations-level stuff that made it impossible to focus on the key parts of my role (feature development etc.). I subsequently turned down a job offer from elsewhere that explained their policy was not to have dedicated DevOps resources for their SaaS platform (devs themselves being responsible for all deployment and system maintenance), and would do so again. Good DevOps people are worth their weight in gold, and at least in many verticals (e.g. those involving payments ) it's virtually mandated that there is a separation of responsibilities between those writing the code and those responsible for delivering the product to customers. I can't see the need for dedicated DevOps resources going away any time soon.
  • baal80spam 4y ago
    I keep hearing that "we are all devops" from my PM. The real kicker is that there is a dedicated DevOps team in my organization, they "just have too much to do already".
  • nrmitchi 4y ago
    > I can't see the need for dedicated DevOps resources going away any time soon. But is what you're describing just.... "ops" without any of the "dev"? I'm not saying that there is not a need for dedicated infrastructure and operations teams at a certain size (and in some industries), but that's not an excuse for devs to feel like they can chuck a new feature over the wall and say "Well, I'm done my job. Please run it and make sure it doesn't break".
    • wizofaus 4y ago
      No silly, that's what the QA team are for! Anyway, our DevOps guy spent probably most of his time "developing" - just not application-level features.
    • judge2020 4y ago
      At a certain point, hiring dedicated Devops frees up x number of developers to continue to develop features depending on the amount of time each developer is spending on performing those devops duties. It’s just another area management can split up job roles to capture more value and allow deeper specialization among professionals.
      • nrmitchi 4y ago
        Sure, but if you're hiring people in a different role that does all of the ops work for developers, this isn't "devops". This is just "ops".
        • wizofaus 4y ago
          Sounds like a terminology issue then, I consider it "devops" because they're mostly writing code that goes into our git repo etc. etc., they still have to do PRs and code reviews etc. But the code is to handle deployments, not to implement features.
          • jen20 4y ago
            Yes: the terminology problem is the is you are using the terminology wrong. DevOps is about cross-functional teams, and has been co-opted by vendors to sell products. And delivery of software is the ultimate feature: without it, nothing of value can be produced.
            • wizofaus 4y ago
              Then everybody else I've worked with over the last 6 or 7 years is also using it wrongly.
              • jen20 4y ago
                Quite likely.
                • 4y ago
    • tamrix 4y ago
      Agreed. It also goes back to the sys admin problem which devops optimised in that the Dev team owns there code in production. I've seen success with a dedicated devops member on a team but having a dedicated devops team just introduces delays and latency when fixing pipelines or releasing. The very same problem we had with sys admins.
    • conradfr 4y ago
      I thougt the "dev" part was because the "ops" were using code to build the infrastructure. In my experience that leads to meh code if the "devops" comes from a former sysadmin, or meh infrastructure if s/he comes from the dev side.
  • deleted 4y ago
    [deleted]
  • dilyevsky 4y ago
    “Dedicated devops” is not devops, never has been. Devops is a culture where, super simplified, you run what you built. “Dedicated devops” is just ops
    • convolvatron 4y ago
      'devops' means I'm going to hire you ostensibly to develop software, but in reality thats just going to be your '10%' time if there aren't any operational fires burning too brightly.
      • dilyevsky 4y ago
        What percentage that works up to is totally up to you and that’s the whole point
      • dijit 4y ago
        “Devops” means build engineering to some Sysadmins who know a bit of python to others Developers who can configure nginx to others. Or a culture; to yet more people. Clearly it doesn’t have meaning if it’s so undefined. FWIW the progenitor of the word defined it as “agile systems administration”: what you’re talking about is the 10+ deploys a day talk from flickr; which doesn’t mention devops at all (despite the conference existing prior).
      • Aeolun 4y ago
        If you are hired as DevOps, or if your organisation does DevOps without any dedicated personnel, and all the devs are now DevOps, you are damn right that everyone'd be fighting all the fires first. The idea being that the people building and running the application are incentivized to prevent fires as much as possible. As opposed to completely separate teams, where the devs have zero incentive to prevent fires. Everyone judges them on features/sprint, so optimizing for that is perfectly logical.
        • wizofaus 4y ago
          That's not been my experience at all. Developers absolutely do need to be roped in to help put out fires when the application is misbehaving, and most of us quite assuredly want to keep that to a minimum. And having reliable, smooth and well-regulated DevOps processes is a huge enabler for ensuring robustness and minimizing the chance of bad code getting deployed to production.
  • CommanderData 4y ago
    Not hiring a dedicated DevOps resource and making your developers do it. Guess what? You've just made your developers do operations. The work doesn't go away just because you've shifted it. I've seen those places too and worked in some, the developers aren't very productive let's just say.
    • pas 4y ago
      If the dedicated guy helps devs write deployment scripts, write monitoring scripts, set up backup-restore-verify cycles, etc.. then it's devops. if the devs proclaim that the devops guy should do it, then it's just the old siloed workflow again. note, that the old flow was not a total shitshow with absolute zero productivity ... it worked for quite a while in many places, but it was bad enough in enough places that a whole "movement" grew out of the recommended solution. it's about keeping the communications/coordination/responsibility-tennis overhead down. sometimes that's best done by saying that you deploy what you wrote in any way you see fit but here's the SLA, and so on. sometimes it makes sense to create infrastructure teams and let dev teams use internal tools to deploy, sometimes this require experts at the team level, sometimes not. and ... of course this can be implemented in the most employee hostile possible way and sometimes in better ways too :)
  • russellendicott 4y ago
    I think what it is is that nobody wants to force more responsibilities and complexity on Developers so they hire DevOps people. Then the DevOps systems are so complex that it needs a ultra high quality engineer to run them. The whole seed of the DevOps movement was that developers needed to do more or the company would fail. Over time, management lost their conviction when developers push back and didn't want to risk losing devs so they "outsource" the devops skills.
• henning 4y ago
  DevOps is sysadmins making unreliable continuous integration/delivery/deployment tools out of bash scripts, YAML files, and webhooks, which are in some ways inferior to things Heroku did 10 years ago.
  • tapsboy 4y ago
    Even thought this is downvoted and it kind of underplays the overall devops role, I understand the sentiment here. That might be a reason why Vercel, Netlify, Render and others are picking up usage. YC has been investing in several similar companies. Big cloud is going in the same direction with AWS Amplify/App Runner, GCP Firebase/Cloud Run and Azure.
  • mountainriver 4y ago
    Heroku was great till it wasn’t, one of the lessons learned from it was we need more compositional tooling, this is a good thing
• c3534l 4y ago
  I think a lot of good ideas in tech get communicated to people who want a cargo cult and don't understand the underlying good ideas until agile becomes a flavor of waterfall, test driven development becomes development with tests, and devops becomes just another wall between developers and their end product.
• tbrownaw 4y ago
  DevOps apparently dates back to around '07 or '08 : https://www.atlassian.com/devops/what-is-devops/history-of-devops https://www.atlassian.com/devops/what-is-devops/history-of-d... (first link I found for "DevOps history") So, how does the current "failed" state compare to how things were before then?
  • nescioquid 4y ago
    From a developer's point of view, it seems to me like the dominance of cloud and containers forced the change to devops. Hitherto your org would have a CM team and a separate group of admins/Ops for your deployment infrastructure. Much of it was manually done (follow procedure docs rather than scripting) and software released on a slower cadence. So at the risk of seeing the past with rose-colored glasses, I'd say things were more straight-forward, less reproducible, slower, and (for developers) simpler.
• iasay 4y ago
  Devops is a failure only when you have the wrong people on the job. Which is nearly all the time because only 1 in 20 engineers has the ability to do both operational and development work to a standard high enough not to put your organisation at risk. If you have several of those 1 in 20 people they will self organise into a methodology which roughly resembles it.
  • throwaway675309 4y ago
    It doesn't help that, at least in my experience, a lot of "pure" software developers absolutely want nothing to do with the ops side, preferring to work on new features and just code, and vice versa.
    • trog 4y ago
      > It doesn't help that, at least in my experience, a lot of "pure" software developers absolutely want nothing to do with the ops side, preferring to work on new features and just code, and vice versa. I have almost the exact opposite problem with some of my team, where they want to spend a lot more time on the Operational side. This is a mixed blessing - they'll learn the depths of various AWS things that are painfully boring to me - but I'd rather they were writing code instead of mucking around looking for some magic AWS solution to solve some specific use case.
      • macintux 4y ago
        > but I'd rather they were writing code instead of mucking around looking for some magic AWS solution to solve some specific use case. Really? To my mind, code is a last resort from a business perspective: I’d much rather developers use AWS features when available.
        • iasay 4y ago
          There’s a happy medium somewhere.
        • pojzon 4y ago
• miked85 4y ago
  "Full stack developer" is a failure as well, which overlaps with DevOps. A very small percentage of developers are capable of this.
  • cratermoon 4y ago
    I've decided "full stack" is just the 21st century was what we called "web developer" in the 20th, which usually mean "one person doing the work of two or three". Except now the expectation is to know a lot more.
    • al_borland 4y ago
      In the 20th century being a "web developer" was a reasonable thing. One person could pretty much do it all without much of an issue, at least for smaller sites. Today, with the countless frameworks and stacks, mounting complexity, and sprawl of code across dozens or hundreds of files... it no longer seems realistic.
  • pojzon 4y ago
    This is a very good point. Companies would love to hire someone doing work of 3-4 dedicated ppl for the salary of 2. As a DevOps Engineer I pretty much am required to know the whole ecosystem of: - JavaScript - Java - Golang - Python - C/C++ On top of that companies often expect you have also at least associate architect for: - AWS - Azure - GCP And ofcourse a security expert that can write OAuth2 implementation in any of the languages mentioned while performing security audits for security certifications!
• cratermoon 4y ago
  On the subject of bringing developers to the table: At a previous employer the DevOps folks replaced the CI/CD tools we were happily using and broke my team's integration testing setup. The new tool didn't support what we needed and had been using successfully. Right at the time we were building out new services that depended on having solid integration tests.
• gmemstr 4y ago
  It's interesting to read the proposed "SoftOps" approach - it's something I've been trying to do in my own little bubble, working on making the lives of developers easier (especially when getting their code from repo to prod), with the developers creating systems. I volunteer for a convention run entirely within VRChat, with a handful of supporting services (API for tracking players, instances and registration, frontends for admin and attendees, and so on), and while the department is named "DevOps" (too late to change it, and everyone knows what we mean), I landed on the infrastructure subteam. While part of the job was building out our new Kubernetes cluster (mostly for scaling. I'm still working on the blog post about this sort of stuff), I wanted to make sure that the developers could be completely abstracted away from _where_ their code was running, and as much as possible _how_. Of course, they're free to mess with whatever tooling I setup for them (mostly Docker images, GitHub Actions, etc), but I've found it very fulfilling to "exist to serve" (because infrastructure? servers?) the developers pushing out code. If we need some custom internal tooling to support that development cycle I'm more than happy to get something written and deployed (something for the blog post). It may be barebones but I wouldn't stop a developer from another team contributing (provided they don't have something more pressing). All this to say... I really like supporting developers in their work, as a sort of meta-engineer. edit: worth pointing out I do something sort of similar at my current employer, but it lands a bit more on the type of devops that the author describes as having failed, and I'm actively working to see if I can find a position that better suites my skillset!
• sto_hristo 4y ago
  I simply quit a job with 2 months after they pinned devops duties in addition to software development. And i've yet to be proven that this can be done by a single person.
  • nrmitchi 4y ago
    I'm curious, in your example, what do you consider "devops duties"?
  • mountainriver 4y ago
    It can but it’s not very practical
  • nixlim 4y ago
    I work in a team of 45 devs. We develop, deploy and maintain our code. Maintenance is done on rotation, everyone is responsible for deployment of their code all the way to prod and we maintain the CI/CD pipelines ourselves. K8s is the only managed service we use on aws. On Azure, we manage our own K8s. It's not "a single person" but as a team, we all do it.
    • hedora 4y ago
      So, all 45 of you are modifying yaml templates and dockerfiles? That sounds like a spectacular duplication of effort to me.
  • ed_elliott_asc 4y ago
    The problem with anything like this where one person has multiple “roles” is that it is hard to excel at one or the other and very very easy to not be good at both. Unless you are given time to spend on both roles independently then you end up cutting corners, it is inevitable as one or the other will be more important at one time and you can’t be in two places at once!
    • trog 4y ago
      > The problem with anything like this where one person has multiple “roles” is that it is hard to excel at one or the other and very very easy to not be good at both. In the immortal words of Ron Swanson - never half-ass two things. Whole-ass one thing.
• hitpointdrew 4y ago
  Meh…DevOps is just System Administration, and Systems Administration is just Sys Ops. They keep changing the title/role but the work remains largely the same. I think is a bit disingenuous to throw “dev” in title, as a “DevOps Engineer” myself I don’t consider anything I ever do “dev”. Ansible is not “dev”, terraform is not “dev”, ci/cd pipelines are not “dev”, helm charts aren’t “dev”. But for some reason companies seem to love the term.
  • dpz 4y ago
    Depends on the job role. I find myself writing more go then anything else...
  • nrmitchi 4y ago
    > But for some reason companies seem to love the term. It's possible that I'm just getting very pessimistic, but at this point I'm fairly confident that companies love it because it makes it way easier to attract candidates and describe one set of responsibilities/position in an interview process, and then bait-and-switch it into what is effectively a systems administrator role.
    • jimmux 4y ago
      I've certainly had interviews like that. In fact my first full time job out of uni was one of those and I made the error (in hindsight) of sticking it out until I could transfer into another role. Now I'm much more careful to screen for sys admin keywords in job descriptions.
    • zero_one 4y ago
      Just happened to me. Now I'm trying to find a way to make an internal transfer happen or decide how long to stick around before applying elsewhere.
  • agumonkey 4y ago
    I think devops means something here, sysops would be about running infrastructure not dev infrastructure, devops would focus on producing dev envs, test envs, CI/CD. Not just setting up the runtime hardware / os configuration.
• pictur 4y ago
  the word devops reminds me of being in constant development and never being able to reach a stable state. constantly new features, solutions, trials, alternatives and other things...
• terpans 4y ago
  2005: your infrastructure is automated using a handful of Bash, Perl and Python scripts written by two system administrators. They are custom, sometimes brittle and get rewritten every 5 years. 2022: your infrastructure is automated using 10 extremely complex devops tools. You automated the two system administrators away - but then had to hire 5 DevOps engineers paid 2x more. The total complexity is 10x. They wrote YAML, TOML, plus Ansible, Pulumi, Terraform scripts. They are custom, sometimes brittle and get rewritten every 3 years. EDIT: to the people claiming that today's infra does more things... No, I'm comparing stuff with the same levels of availability, same deployment times, same security updates.
  • mountainriver 4y ago
    True but to be fair modern infra does a hell of a lot more
  • lox 4y ago
    In 2005 your infrastructure provisioning wasn’t automated. The complexity has increased, but so has what we get. Being able to provision new hardware stacks like software is amazing, in 2005 I had to get quotes from hosting providers.
    • iso1631 4y ago
      Whereas now nobody cares about the cost because it's all hidden away in a massive bill at the end of the month?
  • _vertigo 4y ago
    Not pictured: 2005 your average box served some php and static assets, connecting to some generic relational database. Reading logs means grepping files over ssh. 2022 your architecture runs in the cloud, has multiple flavors of databases, queues, caches, and so on. You have at least two orders of magnitude more complexity because you aren’t just serving a web page anymore - you handle payments, integrate with other services, queue tasks for later, and so on. Your automation may be an order of magnitude more complex than 2005, but it enables two orders of magnitude more functionality.
• draw_down 4y ago
• k__ 4y ago
  The main problem I see with DevOps is that it has become a dedicated profession. If we kept to "Devs should Op their systems themselves" things might look better.
• saurik 4y ago
  > If you look at a DevOps engineer job description, it looks remarkably similar to a System Administrator role from 2013, but... > If DevOps was supposed to be about changing the overall culture, it can’t be seen as a successful movement. People on the operations side of the fence will... As someone who was keenly watching this stuff back 15 years ago, parts of this article connect with my understanding, but the core problem I have is that this article itself is somehow bought into the mistake that led to the failure and so almost can't see the failure for what it is: the entire point of DevOps was that "operations" isn't a job or role anymore and has instead become a task that should be done by the developers. Ergo, if you even still have operations people to comment on it--or certainly if you are somehow hiring dedicated "DevOps" people--you aren't doing DevOps and have already failed. The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps; you simply can't have it both ways, as that's just renaming the same two camps instead of merging them into a single unified group.
  • nixlim 4y ago
    > the entire point of DevOps was that "operations" shouldn't exist, and that operations is a task that should be done by the developers. Ergo, if you even have operations people to comment on it, or if you are hiring dedicated "DevOps" people, you aren't doing DevOps and have already failed. This. My first thought when I was reading the article. Spot on
  • Sebb767 4y ago
    > The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps That's like saying "agile is firing your scrum masters and tell your developers you're agile now". The idea behind DevOps is that applications are provisioned by the people who know the application best, the developers. If everything works out as it should, this gives additional load with creating the deployments, but also removes the overhead when dealing with operations when deploying, updating and debugging - so a net zero in workload, but a gain in the way the application is hosted better and fixing bugs is easier. You still need operations, both for providing the underlying platform (getting a server ready is not a developers core business and it shouldn't be) and for guiding the developers. It should be leaner, but you still need it. Of course, you can also fire all of infra and tell the developers "that's your job now", but that's like calling biweekly deadlines scrum (and leads to equally bad outcomes).
• moltar 4y ago
  I prefer to blur the lines and work with cross functional teams where everyone owns everything. On my current team our front end engineer is able to fix infra, our backend engineer designs and deploys infra. No dedicate devops needed. We use TypeScript for everything: AWS CDK, TypeScript backend, React (TypeScript) for frontend. These are relatively small systems though.
  • clouded 4y ago
    That sounds really fun, because it's not enough to absolutely fry my brain every day fixing obscure bugs or developing new features. I'd like to also fix the infrastructure. I might also be able to handle the responsibilities of a DBA. My wife and newborn son will understand why I can't stop working until at least 7 pm everyday and yell at them because the stress never stops and only seems to be getting worse.
  • oweiler 4y ago
    That is not a crossfunctional team.
  • CommanderData 4y ago
    Sounds great making developers do operations. I hope they enjoy the anxiety of being on pager duty 24/7 too. What a way to burn out your devs to save a buck.
  • tkiolp4 4y ago
    One year later: your end up with a team of burnt out engineers that are about to quit and leave you alone.
• leroman 4y ago
  The worst thing about DevOps engineers having Dev part of their title is that they started to believe they know better than the devs what needs to be done and how. Oh you need a queue here.. Nah you don’t need Kafka.. The Author here is right.. first listen to your customer - the developer
  • saurik 4y ago
    The point of the DevOps movement was to unify Dev and Ops, not rename SysOps to DevOps. The Devs aren't the customer of DevOps, they are the people who are now doing Ops as Ops and Dev were to be merged into a single DevOps, with developers simply coding the operations in the same way they code everything else in the product.
  • somenewaccount1 4y ago
    Lol. It goes both ways.
  • slyall 4y ago
    Whereas in my previous job the Devs has a 2 node EKS Redis cluster for just one key:value. There was also the App with an empty database instance cause it was just a front-end to where the actual data was. Usually whoever is pushing Kafka is the team that doesn't have to run it.
• somenewaccount1 4y ago
  I hardly think renaming it will help anyone. Also, it just sounds like you and your ops team sucked at DevOps, I don't think that's true everywhere.
• lox 4y ago
  My take is different. I think DevOps was wildly successful, most of our infrastructure is now software that can be managed by Software Engineers. The goal posts have shifted, we now have major software challenges where as before we had hardware and operational challenges. Well written tools and cross-functional teams that do both operations, feature work and security are still the path forward IMO, we just need to refocus on developer experience.
  • dilyevsky 4y ago
    +1 tiny teams run infrastructure that previously was responsibility of whole departments at bigco and things mostly work well. Then as soon as they face some minor, totally solvable issues everyone loses their minds.
  • abhishektwr 4y ago
    And now 50% time software engineers are writing infrastructure. Don’t know what is solution but cloud-native landscape has increased cognitive overload.
    • clouded 4y ago
      The point is obviously to pay less people to do more work. What I don't get is when developers themselves are in favor of it like I constantly see with this devops stuff. There's no way they have any kind of life outside of their job. There's no way they have a wife or children, otherwise I simply don't believe for a second they would be in favor of "developers own all the things yay!".
      • moduspol 4y ago
        > What I don't get is when developers themselves are in favor of it If you're comfortable with AWS and have built things as a software developer, it becomes clear very quickly. These things are intrinsically linked, and pretending they aren't is just kicking the can down the road until you have to solve some non-trivial problem. There have been huge innovations and value-adds over the last 10+ years in cloud and serverless, yet everywhere I've worked that silos "DevOps" from devs has already baked in the culture that devs can just avoid knowing anything about AWS, that DevOps will be the gatekeepers, and that devs can just work within the "lowest common denominator" box of tooling that those gatekeepers think is appropriate. Meanwhile, infra costs are skyrocketing but it's all good because we're mostly "cloud agnostic." I don't want to just be closing Jira tickets. I want to actually solve business problems well. And to do that, I don't want to be constrained to someone else's "box," throwing code over the wall to them, and hoping for the best.
• dijit 4y ago
  DevOps was the name of the conference. The actual job title was meant to be “Agile Systems Administrator”. People got the conference confused with the “10 deploys a day” talk from Flickr. DevOps is a failure because it’s a meaningless term that changes depending on the bearers own understanding: the issue is that everyone is right. It’s a nebulous idea. I wrote, rather frustratedly, about this. I even did the research and learned something new myself at the time. http://blog.dijit.sh/devops-confusion-and-frustration http://blog.dijit.sh/devops-confusion-and-frustration Someone on hackernews responded to me once that: “devops is how sysadmins get respect for their role” and that resonated with me. Sysadmins today are what helpdesk was 15 years ago now though.
• oneplane 4y ago
  I don't think DevOps can really be a failure anywhere as it isn't old enough or solidified enough to have a universal definition and any group of activities itself can be mis-performed regardless of what you call it. That includes being a bad webmaster and having a system operations department that isn't able to actually deliver anything. If Development and Operations intersect for some activities, and sharing responsibilities for those activities helps, then it's a win. That includes developers not having to wait for some resource to be provisioned because they are "not allowed to click the button themselves", and operations people not having to do the same "create resource" task all day long.
• smitty1e 4y ago
  Development Integrated Security Container Operations (DevIntSecContOps) is so DISCO.
• wernercd 4y ago
  DevOps is Dead! LONG LIVE DEVOPS!
• tomrod 4y ago
  DevOps is a recent attempt to lower the duplicative and ongoing costs of infrastructure. Here, the saying "you can have it good, fast or cheap, pick one and be ecstatic with two" applies.
• cosmiccatnap 4y ago
  It's so sad to see hyperbolic articles like this saturate HN. DevOps was a stepping stone and it had many lessons to teach us just like the traditional IT before it and just like the roles that will come after it. It was a success some places and a failure others but in general it has been an improvement over most companies processes and to say otherwise is just being salty about how it worked for you at your particular job.
• tomohawk 4y ago
  Devops works for me. I'd rather put time into automation than time into training a crew of people who will fat finger things. Any time I can turn a management or organization problem into a technical problem, its a win. If I can have machines do it, why would I have people do it? I can test what the machines do, and machines can scale a lot better than humans.
• sshine 4y ago
  > All they meant when they said they were “true devops” was that they were doing their own ops work. Why were they doing it this way? > I had spent years trying to bring the developers to the table and own their operational overhead. Is it me, or does the author question his own goal? Aren’t they exactly owning their own operations? > Let’s try SoftOps. Eh, let’s stick to DevOps.
• solumos 4y ago
  > If you look at a DevOps engineer job description, it looks remarkably similar to a System Administrator role from 2013, but with some containers and cloud provider management instead of racking and stacking servers. This is true of many new methodologies/philosophies in software engineering. Many orgs implement Agile in a way that looks more like waterfall than something described by the Agile Manifesto. Remember microservices? Many large orgs that implemented them ended up building gnarly "Enterprise SOA" messes. It seems the old guard will always bend their deeply-ingrained habits only enough to stay relevant. So a lot of the time, that means carrying forward the old habits and diluting the philosophy of newer methodologies.
• tech_tuna 4y ago
  I love that the author ends the post by coining a new buzzword. It's like that XKCD about standards. Also, for those of us who have been around the block, we remember the Old Ways and I for one don't miss all the shitty parts of the Old Ways. Also, it's worth noting: >Since joining Pulumi, I’m seeing the world through a different lense: the world of the developer. This is marketing for Pulumi :tm: - the infrastructure as code platform that frees you from declarative DSLs!
• brundolf 4y ago
  The problem with Docker and friends, from my perspective as a developer, is that it's a whole other skill-set you have to learn, keep up with, and context-switch into and out of. I think that's why it usually ends up being its own job: maybe those technologies bring some order to a huge messy problem-space, but they don't make the problem-space simple. The abstraction is way, way too leaky to use it without knowing and thinking about all the ins and outs of everything that's going on underneath. And you can't even just use the existing knowledge you might have of scripting and systems: you have to learn whole new technologies on top of those. And as a dev, who's already thinking about all the ins and outs of my own whole complex system, that just kneecaps my ability to stay in a headspace where I can get things done. If you want me to do my own ops, they need to have a Heroku level of simplicity. That's what's required for them to not detract from my other work, and not drive me to burnout. Anything less, and I'm going to just keep tossing things over the fence.
  • whakim 4y ago
    I 100% agree with this. I love Docker, Terraform, etc. because it's much closer to the world I usually inhabit than patching drivers on physical servers, but I find it frustrating that once every month or two I'm asked to solve some issue that really requires the specialized knowledge I'd only develop if I was doing DevOps frequently. Instead, I get to the point of realizing there's some issue deep in the AWS networking stack and wish that I didn't have to spend the next day fiddling with subnets (and then not taking away anything tangible to help me solve the unrelated issue that'll crop up in two months' time).
• lkrubner 4y ago
  A bit of humor from an old blog post: Critic: Even if Helm and Helm Charts makes it easy to install a set of apps into a Kubernetes cluster, surely some actions are more complicated than a simple install? What about upgrading a PostGres database? Advocate: Way ahead of you! We worked out that problem a long time ago! You just use a Helm Operator! It’s all really simple! Critic: Really? And this solves all the problems of upgrading PostGres in a stable and reliable way? Advocate: Uh, well, it’s supposed to. It’s, uh, all really simple? Critic: Supposed to? Advocate: Sure, so long as you have a working Go environment, you just use the Operator SDK to generate the scaffold for your Operator. Critic: The scaffold? Advocate: Sure, the scaffold sets up the basics, figures out the permissions, the dependencies, everything needed to install the Helm Chart. Then you can build the Operator container, and install it in your Kubernetes cluster. It’s all really simple! Critic: This sounds complicated. Advocate: Way ahead of you! The good folks at RedHat knew people like you were going to whine about stuff, since you obviously like to whine about stuff, so they created the Operator Lifecycle Manager to make all of this a lot easier. Critic: Doesn’t it seem like we keep piling new technology on top of new technology, to manage the excessive complications of the previous layer of technologies? Advocate: Hey, think of the alternatives. You don’t want to go back to the bad old days of the past, do you? Critic: You mean, the bad old days when stuff mostly worked and I didn’t have to learn 3 new alpha technologies each day? Advocate: That’s a ridiculous exaggeration! Some of these technologies are beta. Critic: And you seriously regard these piles of code, heaped upon piles of code, as an improvement on the old situation? Advocate: Are you kidding? It’s like night and day. I’d rather drink arsenic than get dragged back to the bad old days when I had to write Ansible scripts. We live in the future now. We’ve escaped the old world where every attempt at devops became a painful, confusing, unmaintainable disaster after 2 years. Critic: How long have you been using Docker/Kubernetes in production? Advocate: 18 months. http://www.smashcompany.com/technology/my-final-post-regarding-the-flaws-of-docker-kubernetes-and-their-eco-system http://www.smashcompany.com/technology/my-final-post-regardi...
  • dijit 4y ago
    Add in: “the old way was using bash scripts” and you have every contentious conversation I’ve had in the last decade.
• oweiler 4y ago
  > and having the developers learn and maintain all of the operations practices just isn’t scaleable or feasible. I was on the same boat until I've switched to a company where devs were also doing ops. With AWS, GitLab CI and Terraform, this works surprisingly well. There are still ops persons for the hard parts, but most things can be done by the devs themselves.
• 1270018080 4y ago
  This sounds like people who say cars from the 70s are better because they are only mechanical and people can work on them with their own hands. And modern cars are worse because they're so complex. This, of course, ignores that cars are better in basically every single way now. They're safer, more efficient, have more features, and admit it or not, more reliable. So yeah, pre-devops, managing infrastructure could've been simpler, but it did simpler things. There's an exponentially growing amount of data that servers need to handle. And 2005 tools can't handle it. The industry didn't scale out of it for fun, but necessity. Modern devops is more complicated, but it's better, so, whatever.
  • qaq 4y ago
    "The industry didn't scale out of it for fun, but necessity". Necessity at Google scale is not necessarily necessity for a startup.
• haolez 4y ago
  I already "knew" this, I totally agree with it, but I still open job positions in my company for "DevOps engineer". It's the easiest way to get ops people that are more focused on Kubernetes and CI/CD.
• lr4444lr 4y ago
  Remembering the ideas from DevOps at its inception, the point was to remove the friction between developers and operations: have operations stop saying “no” and developers stop saying “yolo lets ship it”. Well, that's pretty much doomed to fail by design. If you are on the engineering team and not writing features that deliver direct value to the company, wtf are you doing? You're damn right as a dev I expect to "Yolo" it if you're getting paid to make operations safeguarded, streamlined, and resilient to downtime. I don't literally mean I intend to be sloppy, and there are some great tools (esp. on CI and alerting frameworks) that the DevOps culture built, but I can buy that as a service. If you are on DevOps and you are not accelerating devs in the only way that matters to the business (velocity of features with fewer bugs), then what the hell are you doing? Scale should be my problem as a dev to be concerned with because I have deep knowledge of the use case, and should know my stack. I'm sorry to sound harsh, but IME, DevOps people have a terribly inflated view of their contribution to a company's bottom line. Devs are an awfully scarce resource, and these guys are often more than capable of producing real value to drive a business, but they seem hell bent on spending it on derivative, second order concerns that just aren't really that important to employer.
• pojzon 4y ago
  DevOps is a culture of work. Same as some ppl waking up every day to run a mile, then drink an energy drink and drive their kids to school -> Being a real DevOps takes effort, commitment and rigourous routine. After some time you get used to it. But not a lot of ppl can actually get to that point. Someone asked me how to grow DevOps culture in a company -> it's like asking "How to make ppl run 1 mile every day?". You cannot "make" ppl do that, you have to find correct ppl. And here is the reason why DevOps as a culture failed in the real world.
• 734129837261 4y ago
  Eh, I can make a nice and similar argument against "full-stack" developers. I'm one of those, but I specialise in the front-end. Because that's what I'm really, really good at. When I see "full-stack" engineers do their thing in the front-end I almost always spend an outrageous amount of time on their pull requests. No semantics, unnecessary CSS, no accessibility features, not using the right tag for the job, not familiar with browser APIs, not familiar with cross-browser concerns, nesting tags that shouldn't be nested, not familiar with paint/composite/layout and other performance tools, using `grid` where they should use a `flex` and using 3 layers of nested `div` tags where zero would suffice. If they suck this bad at the front-end, as a supposed "full-stack" dev, I can only imagine how little they also know of the back-end, let alone the operations side of things, not to mention testing, CI/CD, etc. Hell, can I even trust them to sign off on PRs?
  • tkiolp4 4y ago
    Blame the companies that hire fullstacks.
  • stavros 4y ago
    > I'm one of those, but I specialise in the front-end. Because that's what I'm really, really good at. What if they specialise in the backend, because that's what they're really, really good at?
• andrewallbright 4y ago
  I've really enjoyed learning about all aspects of creating software. Whatever we decide to call it, I'll always enjoy doing things that don't always fit into a neat little job description.
• crikeyjoe 4y ago
  Devops is required if you want deployment portability. Cloud providers don't like that so much.
• tkiolp4 4y ago
  Tired of working on infrastructure stuff while holding the “software engineer” role. Give me product features, bugs, interesting and boring product problems to solve. Keep your yaml, k8s, gitlab ci/cd, terraform, on call rotations. Or better, hire a dedicated infra engineer to handle all that stuff.
• tacker2000 4y ago
  I think the overhead here is not devops itself. Docker and the tools like ansible that have evolved until now are really much more useful than the scripts, etc we had back then. The overhead is paying thousands for AWS/Azure/whatever cloud services for your little internal company server or small SaaS when you could just get some on prem servers and host it yourself. A lot of people have been sold on proprietary, fancy stuff like ec2 and lambdas that lock you in just to suck more money out of you. Of course, failure redundancy should be also worked into this, but even hosting in 2 locations is much cheaper than feeding the hungry cloud beast.
• dsr_ 4y ago
  Trying to do DevOps without any [sysadmins, network engineers, DBAs, security specialists] is like trying to write an accounting system without any CPAs, or manage a chemical processing plant without any chemical engineers. Reading the books as a substitute for experts with experience only works a little way. The point of DevOps was to take that experience and multiply its effectiveness through the tools of modern software development.
  • hedora 4y ago
    In practice, modern DevOps is more like having chemical engineers set all the knobs at one plant. Instead of reading the books or hiring more chemical engineers, you have a computer copy the knob settings from the old plant to the new, unrelated plant.
• kmbfjr 4y ago
  The “culture” and “movement” was marketing by someone selling conferences and books. “What is your devops culture” has to be one of the most asinine questions of the last 10 years. It all described a way to get to continuous deployment and IaC, all of which suffers from entropy the moment someone decides that is how it’ll be. None of the tools available to do this lofty goal are are really up to the task. There is good, there is bad. A failure it is not but it is not a success either.
• lloydatkinson 4y ago
  A bad devops team becomes what I call "dev obs" which is short for a phrase I made up, developer obstructions. I use this to describe devops teams that have forgotten (or maybe never knew) their primary purpose and instead do their own thing independent of the developers and business they are supposed to be supporting. This can include: Upgrading, replacing, deprecating, turning off key parts of infrastructure without any notice or warning "Scheduled" changes to systems that are scheduled nowhere but their own minds and then become angry when this is pointed out Turning off logging because "it's expensive to store logs" (get a better logging system then) totally ignoring the huge costs of production failing with no visibility as to why Enforcing their own ideas of unit test coverage and the like without speaking to the developers, as if they know more about it than the developers Enforcing package management security in a way that simply isn't compatible with a particular language's tooling leading to the scenario that all deployments, even security hotfixes, cannot be deployed without needing to escalate and get into arguments Taking a generally antagonistic and hostile approach to communicating with the developers rather than again supporting them
  • nrmitchi 4y ago
    > Taking a generally antagonistic and hostile approach to communicating with the developers rather than again supporting them This point is very ironic given your entire comment. It sounds like many of your complaints may be due to you not understanding the requirements that another team may have. You're complaining about costs and requirements that you are trying imposing on another team ($$, lax security). As an example, logging can get extremely expensive. A "better system" can't always fix an ungodly amount of unnecessary data. It might be best if you try to understand why they are working the way that they are. Trust me, they are not doing unnecessary work just trying to fuck with you.
    • lloydatkinson 4y ago
      I’ve never complained about costs, read again. Strange you didn’t comment on any of the other points but decided the entire argument was incorrect because of two points.
• armchairhacker 4y ago
  At a certain point operations becomes development and vice versa. When you really need to be good at managing computers and servers, you have to understand how those things work. At a certain point you're literally writing shell or even Python scripts. When you need to be really good at programming computers and servers, you need to understand the underlying OS and hardware, for maximum performance and to solve particularly-annoying bugs. At a certain point in optimizing your development and configuring your code package / repository / etc. you're basically doing ops stuff. Of course there are people who focus on developing and not really managing their systems, and people who go deep into managing the systems and not really learning to code. But I would imagine any half-decent ops person has at least basic knowledge in software development. And while there are people who can truly be both full-time developers and full-time operations, they probably don't do both in their job.
• adfjalkfja 4y ago
  Big part of my job is convincing the bosses we don't need X new shiny thing... we still waste years chasing random tech that doesn't benefit the customer at all though
• donutshop 4y ago
  A lot of the DevOps engineer I know are more like resume driven YAML engineers.
• strangescript 4y ago
  Just use the cloud, honestly. "No, but there is this weird thing we do, and need custom yada yada", no just stop, conform to the cloud, stop being a snowflake. "Its so expensive!". No, its still cheaper than on prim and paying dedicated teams to manage it. Just use the cloud in a sane way. Teach your devs with the infinite guides and docs freely available online. Your life will be easier. If your devs aren't interested in your infrastructure, they are bad devs.
  • schipplock 4y ago
    > If your devs aren't interested in your infrastructure, they are bad devs. so, 90+% are bad devs? :D
    • physicsguy 4y ago
      It's the same as many other things in software - if all you care about is writing software, then yes, you are not a good developer. You need to have an appreciation for the environment it runs in, interfaces between you and other teams, the key business requirements, the constraints it's under, etc. etc.
• deleted 4y ago
  [deleted]
• myrryr 4y ago
  predev ops, I had to put in paperwork for servers, they would come back not set up correctly, we would have test and prod being different, it would take a long time, and it would cost us a lot. post devops, I put in the scripts, and we use that for deployments. The ops teams review the scripts. I don't know what failure looks like for other people, but it is a LONG way from where I am standing. We put in CDK scripts in as part of our development now, and it works pretty damn well. The different teams who want to review different parts do so. Deployments are fast, and more importantly, they are the same as the dev deployments, which we don't have to go though months of trying to get set up and running. We have a capped dev AWS account, and that solves a LOT of issues.
  • Riverheart 4y ago
    Just curious, IaC scripts committed to code repo or separate repo?
    • myrryr 4y ago
      Depended on the project, if the project was using Lambda functions etc, then it was in the same repo. But typically the projects made docker images etc, and how they were to be deployed was not important to the containers themselves, so they ended up separately. In part because we wanted to opensource the containers themselves.
• hintymad 4y ago
  Why is DevOps is so hard in other companies yet so easy in Netflix? I mean, the entire monitoring stack of Netflix was automated by one person and managed by four. Two people in Netflix used to manage their entire messaging pipeline, including suro, Kafka, Zookeeper, and some Druid stuff. Their Hadoop ecosystem used to have a handful of people and their end-to-end ingestion pipeline that aggregates and demuxes data into Hive with 15-minutes of maximum delay was written and maintained by a single person. Their key manager, well before Amazon KMS existed, was implemented and maintained by a single person. Their engineers in the cloud platform team were oncall 24x7 yet they only occasionally got paged. Oh, their cloud platform team had fewer than 20 people too. Their entire Cassandra team used to have fewer than 20 people. They predictively autoscaled their clusters with a system created and maintained by merely 2 or 3 people. According to their engineers, they did a few things on top of the excellent foundation of AWS, and they just did them without any fuss: - API-based full transparency. If there's a function, there is an API. Anything you can do is explicit in API. - Powerful monitoring support so "instrumentation to death" is not just a slogan. - Decentralized control. For instance, teach team has freedom to decide how to load balance their traffic and how to handle unresponsive nodes. - Assuming everything can fail and build mechanisms to account for that assumption. Chaos engineering. Autoscaling from get go (again, thanks to Amazon for such a powerful feature from day one in EC2). - A shared culture: don't tell me to learn your shit. So, no friction from handling shit like Puppet, like Chef, like Terraform, like HCL, like whatever yaml or DSL that folks on HN passionately advocate. Like learning how to embed a jinja template in a job description with 5000 lines of Ruby code plus some system-specific half-assed DSL just to update an environment variable? Never gonna happen. Don't get me wrong: they are probably nice technology. They are just too damn low level and irrelevant to most engineers. - Instant gratification. If I make a config change, I want to see it in production in seconds, safely. If I make a change in my code? The change will deploy with all the guardrail in seconds. So, a puppet change that takes on average 15 minutes to materialize? That's just garbage. - No surprise. So, a Chef script goes behind my back to update my OS and screws up my production service? Never happens. Immutable infrastructure was implemented from day one. So the question is, why are those thing hard in other companies? Do engineers enjoy getting paged at 2:00am? Do they enjoy spending at least 1/3 of their time handling so-called operations? Or do they enjoy writing rants like DevOps is a failure?
• ankushnarula 4y ago
  Every time we standardize mediating layers of leaky abstractions we are creating novel territory that accrues side-effects. Once there’s enough people working to manage these side effects, experts, products and services will emerge to promote the legitimacy of this new layer. There have been many attempts to solve this by revisiting first principles, but most of us are oblivious and/or too vested in one of these layers or too averse to disruptive changes. Hence, when we run up against bottlenecks and side effects we deploy more leaky abstractions and the cycle continues.
• hedora 4y ago
  From a developer perspective, DevOps is when you fire all the competent operations people, and then force developers (that chose not to be in ops) to operate the code they wrote. This burns out developers (bait and switch tends to do that), and with silicon valley's standard 4 year vesting cliff, the people that wrote the software just quit and go elsewhere (or get promoted out of the team) about a year after whatever system they built hits prod. Now (since hiring competent ops people isn't shifting left or right or whatever), you need to hire more systems developers, and pay them hazard pay to operate other people's abandoned garbage. (It is garbage because the previously burnt out group of systems engineers was constantly distracted by things they do not care about. They were constantly distracted with prod issues because they are bad at ops.) I've worked in devops teams, and with competent operations teams. The latter ends up creating a much higher quality of life for the developers and ops team and also a better product. In related news, hiring an optometrist to clean your teeth is a bad idea, even though they went to med school.
• partiallypro 4y ago
  I was DevOps at my last job but accepted a new job just as a developer. When I was DevOps, I was also system admin, SecOps, etc. I was always on call and had no real backup. Luckily most things I had set up just worked, but honestly it wasn't worth the stress. It wasn't consant stress, just a sudden surge of stress during certain deployments or problems.
• contingencies 4y ago
  To make error is human. To propagate error to all server in automatic way is #devops. - @devops_borat ... via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup
• throwaway787544 4y ago
  I agree. > the anecdotal evidence I’ve gathered has been that the conference are heavily attended by operations, and less attended by developers. Most Ops people - fuck, even most actual DevOps Engineers - have no clue what the fuck DevOps is. They (rightfully) assume it's just a trendy new word for the same old Ops bull, but in the cloud and with Terraform. DevOps failed because it never educated anybody except a very small handful of people who actively were looking to solve big organizational problems. It was too many things to too few people. It could succeed only if you brought everybody in the entire org through three different training courses. And that's because DevOps tries to make Operations uplift literally the entire technology organization. DevOps is dead. The ideas are great, but we need to bring the ideas to people outside Ops. Until then it will just be a slightly-more-technologically-advanced Ops. (disclaimer: I am a DevOps Engineer that hates the fact that this is my title)
• StopHammoTime 4y ago
  This is a hot take. The reason that you need to do things the ops way is because ops knows how to run applications in production. There's a reason the meme "worked in dev, ops problem now" exists. You need to meet all of the requirements of an app that's running in production from a technical, availability, security, and policy point-of-view. It's not easy and that's why this will never work. Software is hard, it's just that a lot of developers used to cut their code, run it on their laptop, and let someone else worry about it. It's different these days (although not as much as I'd like). We don't make you use these tools because we want to, we use these tools because we're required too. No one cared about ISO27001, SOC2, or PCIDSS compliance for your crappy PHP app you ran on your cpanel. They didn't care back then you were using md5 hashes to "secure" passwords. The world is fundamentally different to what it used to be 10-15 years ago, and the requirements from business are astronomically different. Edit: and to people saying "oh you could just run it on a single server", no you can't because certifications like ISO27001 require certain levels of availability and DR. You're not going to be able to guarantee that with a single server running in a rack somewhere.
  • deleted 4y ago
    [deleted]
  • dhzhzjsbevs 4y ago
    > This is a hot take. I'm assuming you mean your comment, not the post itself. > The reason that you need to do things the ops way is because ops knows how to run applications in production Stability in production is one metric. Ops overindexing on this metric is exactly what causes the friction with developers. Developers are trying to ship value to customers. Uptime is only one part of that equation and for most businesses, it's not even a very important one. The author points this out near the end. DevOps can't convince devs to use ops techniques if all the reasons for using those techniques are based on the flawed assumption that development velocity isn't important.
    • gonehome 4y ago
      > “Developers are trying to ship value to customers.” I’ve also seen this be fairly rare. Devs shipping nothing - not even aware if what they're merging will turn on. They write something they haven’t really tested, merge it, and call it done - a user may never see it and they don’t have any knowledge about how the thing actually gets built and shipped. Obviously this is worst-case, but in my experience this is a common default. The complaints about friction are because they’re actually forced to reason about how the machine works in order to ship something beyond merge.
• nathants 4y ago
  excellence in software engineering isn’t something you can buy. a university can’t bestow it. a title or certification can’t assert it. it’s something one grows over time because they are passionately curious. it grows because they love to explore, to discover, to understand. there isn’t a product that can be purchased and consumed that makes you thinner and fit. software engineering is no different. the only way to grow is to exercise, and the best way to exercise is to love doing it because it is fun. manipulating metal, cloud, laptop, browser, and server are not very different. all can be made dumpster fires. all can be made magnificently simple, or magnificent in some other aspect. all are discoverable, and easy to experiment with. all are at your fingertips, cheaper and more accessible than ever. DevOps is a product, best to avoid the purchase. devops is a fantastic idea. one should be good with infrastructure, because that is where code runs. one should be good with code, because that is what runs. one should be able to reason about their systems from top to bottom and inside out. how else could one simply intuit what is wrong, and make it less so? excellence in software engineering is about seeking out that which is ugly, and making it beautiful. seeking that which is hard, and making it easy. seeking that which is impossible, and making it hard. why would one accept any arbitrary boundary, and think that this reality applies only on one side of it? where we’re going, we don’t need roads. we need curiosity, imagination, and fun.
• nunez 4y ago
  as a long-time consultant in this space, i don't fully agree with this take. the entire point of devops was to make devs more ops-y and ops more dev-like, i.e. everyone is a developer. to some extent, the movement has been successful on this front. devs generally have a better idea of how to run platforms, and ops generally can write enough code to kind-of support them. tooling like terraform and kubernetes have helped a lot here. there are many companies that have definitely lost the plot (by forming devops teams that are glorified cloud admins or release engineers), but i'd say overall the bar is higher now. a lot of sysadmin jobs require shell experience and Git, which was definitely not the case ten years ago. the real reason why the culture part of devops hasn't "happened" is complex. in many bigcos, devs and ops are completely separate business units with different corporate agendas and kpi's. dev teams are also restricted from admin'ing their own stuff in production bc of audit risk, so even if they wanted to, it's an uphill battle. however, dev teams at these companies are by large incentivized by what they ship; ops is not as high of a priority. what i've seen a lot of lately are ops teams becoming platform feature teams with dev teams as their customers. many ops teams have embedded into dev teams like consultants to help them move stuff onto these platforms, and they make admin as hands-off for themselves as possible. devops is to thank for all of that. before "devops," zero-touch platforms were the hotness for ops teams, but they didn't have the chops to write the code needed to do that, so they became huge exercises in COTS integrations. i do agree that devops is very infra-biased, /r/devops is /r/syadmin redux, and that forming a "devops team" requires serious questioning
• vegai_ 4y ago
  Not a problem. We just call it "cloud architecture" now.
• Aperocky 4y ago
  I've always looked at DevOps as Development absorbing Operations but never through the lense of the other way. We develop, we ship, we maintain. Never had any real problem with it, we manage the infrastructure by writing a lot of yml files, some of them compiled by templates. Anyone who wrote software is expected to also write the infrastructure that supports that software. And the team as a whole take care of operational duties. I've always thought that was devops.
• benjohnson1707 4y ago
  Isn't SoftOps just product thinking from an ops perspective / designing the developer experience?
• Aeolun 4y ago
  I agree that DevOps is a failure, but not for the reasons that OP states. Or maybe exactly for the reasons OP states. People that would traditionally be called Ops were turned into DevOps overnight, and felt like they suddenly had the mandate to try and barge into development, offering true, but ultimately misguided help. If you haven't asked me what I need yet, don't bring me a platform that is "going to solve all my problems". I imagine this is why the developers in OP's story do everything themselves. At least they're able to fix something if it breaks, instead of having to rely on the ops team that is off chasing the next shiny thing.
• durnygbur 4y ago
  Both DevOps and Front-End hysteria are enormous. My conspiracy theory is that FAANG+MS does this to burn down resources and prevent anything interesting created outside of their sphere of influence. In one workplace I was barely keeping my head over the surface in the Angular+NGRX+Observable cesspool then started to talk about corresponding infra and the backend person was all like "let me configure and provision an autoscaling geodistributed self-healing supercluster". KILL. ME.
• jillesvangurp 4y ago
  There have been a few clear changes over time. It used to be that you delivered software to the ops team who then took it upon themselves to create servers to run that software, package up the software, testing it, etc. Delivering software usually went in the form of an email suggesting that this or that person might want to put software on a server or use some pre-built artifact. These days software teams deliver ready to run software; usually in the form of some containerized binary produced by a continuous integration environment. No ops team is involved with deploying that container either because the deployment is fully automated. The CI/CD environment is typically provisioned via SAAS. We use Github actions for example. Our deployment process is "merge to the production branch". The role of ops has been reduced to providing the automation that does several things: - provisioning the infrastructure, typically via some IAAS platform like AWS, GCP, etc. with some automation (terraform, cloudformation, ansible, etc.) - automating the deployment of software on that infrastructure. This normally is some kind of one liner in a CI environment that triggers the right scripts to run. - automating the building of the software - dealing with backups, monitoring, etc. For small teams that work is easily handled by a specialized person for whom this isn't even close to a full time job. I know because I'm that person but I mostly do other things. In fact, I get really grumpy if I have to spend time on this stuff because it is a bit of a time-sink and it blocks me from doing more interesting things. Ops teams still exist of course but they are employed by IAAS providers or companies that still run their own hardware. They don't get involved with how software is packaged and deployed anymore. They just keep the infrastructure up and running. Most senior engineers know enough to automate most of the above. And becoming a proper devops engineer just means stepping up and learning how to do all of it.
• CraigJPerry 4y ago
  I disagree, i move in different circles from the author. In my little world DevOps is often more dev-heavy (and needs to mature it’s ops credentials), SRE has become the new-age Ops. DevOps is primarily tackling a people problem, there’s certainly plenty of useful tools to help but at its core, it’s about people. Encouraging people to work in frequent small increments (think XP) rather than quarterly releases. Getting rid of the change management beauraucracy, encouraging developers to consider security as they’re designing and writing software. DevOps is a broad church, it’s effective too. People like to over-focus on the tools of DevOps but I’d take today’s world of version controlled (vs. “Ohh Brad has the config GUI code on his desktop, speak to him about making a change“), automated CI pipeline (vs. builds at the end of the quarter that require stop-the-world help and assistance from all developers, also RIP merge guy), automated deployment into many environments (vs. Excel list of tasks for QA team and different excel list of tasks for prod ops). I’ll take SOA over the-DB-is-the-network 00’s enterprise software architecture, oh and with automated DB deployment (albeit still without a real solution to automating stateful db rollback even today). These are all DevOps factors, from architecture to testing, from security to working and growing together as a team. The tool-only view of DevOps is just yet another excuse to ignore the hard problem: helping large numbers of people work productively together.
  • bovermyer 4y ago
    It appears that you and the author both agree on the core definition of "DevOps" as a term. Mr. Briggs' article has two main points: * convincing developers to do operations is too hard, and * what they actually need is another type of engineer to build and manage an operational platform for them With regards to the second point, that already exists: platform engineering. The first point is messier. I disagree with the point, but it also misses the point. Like you say, the real goal is to help large numbers of people work productively together. That's a nebulous goal, though, and difficult to guide people towards. I think that's why people are confused and frustrated by "DevOps" even this long after its conception. It's also why people will continue to be confused and frustrated by it long after the term itself has gone away.
• jhoelzel 4y ago
  I would like to point out: DevOps is a failure ---- where you work and how you see it. Just because you play bs bingo for naming things does not change your internal processes and no i dont mean the "culture" but the need for the term "SoftOps" because they are "for developers" just explains to me that your org probably has a huge communication problem and that is bad to begin with. Look we use terms like "DevOps" Engineer to explain that we do all things around the stack, but serving the devs is only a tiny tiny part of this. First and foremost our dutys are for the CTO and the Vision. Now the CTO should and does appreciate and use the input of his team to perfect the vision but you get the point. OPS isnt there to serve you or limit your spending either, OPS should take care of compliance in all areas needed and to keep the operation of the org running. DevOps ___for me___ means that the person I am working with can handle the software in the context its being brought up. That includes people operations just as much as the workflows we put in git. It should have been the next level of the Senior Programmers, but since we are truly lacking Programmers that actually do unterstand stack, code and sourrounding requirments, we foget what the term really means. "DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality.[1] DevOps is complementary with Agile software development; several DevOps aspects came from the Agile methodology" (wikipedia.com) its not a synonym for "dude who sets up your cluster"
• bof_ 4y ago
  > Reading logs means grepping files over ssh. Ah, happier times.
• revskill 4y ago
  I have no issue with stateless services, they're simple to work with. The issue is with relational database. They're trickier to work with. I'm asking myself, is the whole devops is to simplify db migration process, or to complicate stateless service ?
• deleted 4y ago
  [deleted]
• acd 4y ago
  I think the point is that programmers implement automation and operations. Ops cannot fix program bugs. Programmers need to think about monitoring, automate deployments. Kubernetes and microservices solves some problems but adds complexity. Distributed tracing vs local debugger. Lots of yaml. Kubernetes Network abstractions overlays where pod network is non reachable from dev machine without proxy etc.
• bcoughlan 4y ago
  "DevOps" became a job title because the level of knowledge required is enough to warrant one. DevOps Engineers need to know their stack well enough to respond quickly to production issues. It's not enough to live at a high abstraction layer when you're responding to non happy-path events. On top of sysadmin and networking knowledge DevOps-ers need to know a bunch of specialist tooling (Ansible, Terraform etc.), need to know the ins-and-outs of their cloud provider AND learn Kubernetes which is an extra abstraction across cloud providers and comes with its own complexity and quirks. All this does make me wonder where we've come from ten years ago when we thought of Docker as a useful. Is it really easier to run a backend now?
• AtNightWeCode 4y ago
  I believe a large part of the devops culture came from the early clouds. It was difficult for ops to fix anything without involving developers. And for developers to troubleshoot basic things like network issues. This because the poor quality and visibility of the cloud services. The clouds are much better today and requires less skill to use. The unfortunate consequence of this seems to be that more ops tasks end up being done by devs.
• nkotov 4y ago
  Every company has a different definition of what DevOps is and the success metrics surrounding it. I think the main principle should be though that you help developers move at their own pace. The way you do it will be different at every company. Some resorted to creating internal tools or using something like Backstage. Others literally forcing developers to learn AWS and Terraform so that the developers know exactly how their applications run. I think nowadays, there is more benefits for developers to have experience on how their application get built, deployed, and ran.
• welder 4y ago
  Everything is good and I like the new term SoftOps, but I need more examples of SoftOps in the real world or to me it's just DevOps with a different name.
• knite 4y ago
  Systems administration and operations are tightly scoped. DevOps is, in my opinion, an umbrella term representing all non-core engineering work: servers, build pipelines, on-call response, infrastructure, and more. The tricky part is that a lot of these areas are relatively new, and the older ones like server admin have changed dramatically over the past two decades. DevOps hasn't failed. DevOps is in its infancy and is going through technical, strategic, and philosophical growing pains.
  • ryathal 4y ago
    DevOps has pivoted I think from the more original goal of having Devs do a lot of Ops things. This isn't really a viable solution as almost every compliance system mandates the people writing code and the people deploying/running systems are different. What we have now is Ops that use things like source control. Write scripts that are more modular, reusable and composable. A new application can have cloud resources created and allocated in a day rather than a month of tickets to several other teams. This is also granting visibility to things that Ops may have been doing, but it was scripts on a PC or build server that only Ops had access to.
• Too 4y ago
  Mechanical engineers had DevOps figured out long ago. They call it DFM - Design For Manufacturing. It doesn’t mean every designer has his own metal sheet press on his desk. It means he understands his component is used in a bigger picture, should fit into the robot assembling it, use the same type of plastic and standardized screw as all other components, and so on. Same goes for software, running your own shit sounds romantic and all, but having every developer in your company picking their own monitoring stack or understanding the intricacies of kubernetes networking is really not effective. Some central guidance defining the standard screw head to use is needed here as well.