10 ms·

2023 DevOps Is Terrible

116 points by moonabid786 3y ago

jlbooker 3y ago
This return a 404 now. Anyone have a better link?
- MostlyAmiable 3y ago
  Apparently it was plagiarized from here: https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88162c86d7 https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88...
hinkley 3y ago
The fact that we have a “devops team” at work infuriates me. It’s enough to distract me me from how “Scrum” has destroyed Agile.
- 0x008 3y ago
  What is so horrible about it? Our devs are happy they can focus on fixing unit tests in pycharm, while others care about where the code is going.
  - pjmlp 3y ago
    Because that is the software developer/systems engineer division that already existed 20 years ago. The idea behind devops was to be a single team.
    - bojo 3y ago
      > The idea behind devops was to be a single team. I assume you are pointing out that split-teams is the insane part, in which case I agree. The philosophy was supposed to define the blurred line between developers and operations, using tooling to help facilitate the shared responsibilities where sensible and make hand-off a breeze. As you pointed out, the division existed 20 years ago, and it still exists now at many places. That lack of integration doesn't offer much improvement in relation to the past.
      - ddaaeugzzgg444 3y ago
        [dead]
  - matsemann 3y ago
    Because it's just a hip way of saying you have a sysadmin team. How can something or someone be "devops" if all the developers are in one team doing dev tasks, and the ops in a separate team doing ops stuff?
    - dijit 3y ago
      If we keep reinventing the same paradigms; isn't it a perfectly fine paradigm? It's unfortunately inherent that some people want predictable, scalable, observable, stable- and at the same time they want fast moving and many new features. I think it's not so much that dev and admin work is terribly difficult, it's more that "a man cannot serve two masters" and someone will be focusing on stability and someone else will be focusing on quickly deploying. Trying to do both yourself I think leads to burnout.
    - root_axis 3y ago
      The term is meant to encompass the "infrastructure as code" workflow. The "dev" in "devops" refers to the skills necessary to read and write code that manages the state of the system infrastructure.
    - ecshafer 3y ago
      A devops team is in practice typically very different than a sysadmin team. Good sysadmins automate the hell out of everything, but it’s usually not something other people can use. Devops teams automate all of that stuff but then they also let teams build new infrastructure when they do deploys and automate that whole process, but it’s client facing. It’s development and operations, it’s a pretty good name, though I think platform or production engineering is typically a better name.
      - hinkley 3y ago
        The quality of devops scripts, from the perspective of a person who likes to write code that has to work right the first time, is about two standard deviations better than sysadmin scripts. But still of a par that would keep someone from getting promoted past a Senior Dev title. What’s worse is they can be condescending while shoveling out absolute garbage. If I’m honest that’s the part that bugs me the most.
  - medellin 3y ago
    The thing i have come to dislike is that in both companies i work for it lead to strange working relationships between ops/eng. anything that breaks in the deploy pipeline is now “the devops team problem” and they normally act like it’s also your fault. In the past this has meant people messaging me saying they are blocked because their dockerfile fails in CI. Then you walk through it with them and it’s also broken locally and it comes down to them pinning to something like latest tag and the image “shocker” changed. But now you are fixing it and after you go remind everyone that you should pin your image to as strict as possible image tag. But no one cares because they can move fast and break things and when it does blame it on CI and now it’s not their problem. Anyway this is just one tiny example but the same underlying thing starts happening in all parts of the development cycle. “My deploy doesn’t work” … well your program has an error and crashes on startup but looking at the logs would be too much work. Let’s just say we are blocked and log off. As you can see this is a culture issue but it seems to be common when you have a devops team. And even more common when management (read senior management and up) think that developers “should just be developing”. Anyway thats my rant.
    - anotherhue 3y ago
      I have seen infra/devops scapegoated too much. Half the time the developers in question won't even read the error output before raising a ticket. The only solution I see is to enforce commit checks before hitting shared environments, but that comes with its own set of complaints.
      - tstrimple 3y ago
        > Half the time the developers in question won't even read the error output before raising a ticket. If they had no DevOps team to raise a ticket with, and it had to get done they would find some way to figure it out if they had the ability to modify and initiate their own pipelines. The reason I like "you build it you run it" is I feel like I have far more control over my own destiny. You're not having to negotiate and coordinate with a separate team through some ticketing system. Far less wasted back and forth and miscommunication. If that means you have to expand one of your teams with an additional DevOps person to foster understanding of the tooling on a team who hasn't done it before that's fine. But don't make them the DevOps person. Their goal is to enable the rest of the team to pick up these skills. Their goal should be to transition into a normal contributing member of the team working on product development because the team no longer has a need for a DevOps person.
  - steve1977 3y ago
    That’s fine, it just has nothing to do with DevOps then. The whole point is that these are not separate teams.
  - Twirrim 3y ago
    So in other words, you made silos. More throwing things over the wall and having it be someone else's problem. That's fundamentally the antithesis of DevOps. People do not fix the pain they don't see, and managers doubly so. By shifting operations off to a "devops" team, you just end up with developer teams that don't know about, and don't care to fix, the things that make it painful to deploy or run their code in production. Those that have to deal with the pain rarely have any political leverage they need to even enact change. There's zero motivation for either the developers writing the software _or their management_ to fix it. Dev manager incentives are purely aligned with getting code and features built. To give a nice real world example, building out new regions for AWS services used to be a pain in the arse (since I left I hear some major top down initiatives have fixed that). The service team I worked for, almost doubly so painful for region build. Standing up an entire region was something like ~50 days of engineering time. Not because it was especially hard, per se, but because the ops team always had to wade through piles of refactored shit that introduced new circular dependencies, or wasn't documented, or broke other aspects of any automation we could produce. (I remember one DynamoDB schema overhaul, where we did a complicated migration. Next region build, service components wouldn't stand up because we'd used the automation to create the new table schemas that had replaced the old table schema entries, but the service code still expected to see the old schemas in the right format. It took several months to get the dev manager to agree to engineers spending time cleaning up old schema references from their code base.) The dev manager incentives were not aligned with making it easy. It wasn't their engineering resources being consumed, and they had features they needed to ship. The developers never changed the way they worked because even though they were told the issues (and even though they were, virtually to a person, great people to work with, considerate etc.), they never experienced the pain so they never thought about those consequences of what they were doing when they were doing it. Ops manager never had any leverage, because every single time we managed to get a region launched in time as to not be an issue, and every time things were down to the wire it was clearly as a result of the ops team not building the right automation :eyeroll:, rather than that the dev teams were making some kind of brand new nightmare each time. It wasn't until the ops team had lost enough members of staff to that BS, and region build shifted on to the developers themselves, that suddenly most of the problems went away, never to appear again. They were aware and conscious of the pain certain decisions would cause because they had felt it themselves, and they made sure not to introduce more pain. DevOps is about making sure incentives and politics align to make things overall better. Shoving it in to a silo is literally the complete opposite.
  - stavros 3y ago
    It's like saying you have a self-service waiter.
- tstrimple 3y ago
  I think there is room for a DevOps team if they are focused on creating and maintaining the overall DevOps platform. But if they are taking tickets and setting up pipelines for individual applications, they are doing it wrong. But a team focused on making a streamlined platform with ready to adopt patterns that already account for compliance and security related concerns, they can spread DevOps adoption more effectively and consistently throughout the organization.
- whstl 3y ago
  The only thing I like about the return of sysadmins and the end of the "you build, you run it" culture of Devops it is that I'm 100% off the hook if something dies in the middle of the night. The so-called DevOps guys are stressed out of their marbles, but when the CTO calls me 4 AM I can't say anything but "Sorry dude, I'm not a clairvoyant. I have no permission to do shit in prod, call Ops or give me AWS access now". He stopped calling. The fun part is Ops wants us engineers to help more, but they don't even trust (and some don't know how to run) my Terraform or CloudFormation scripts (I let them choose which), so they just read the tf/yaml and click AWS buttons manually. Often making lots of mistakes.
  - hinkley 3y ago
    I always do a better job of writing logic when I sympathize with the victims. Some of those late night calls are devs setting ops up for failure. It was meant to close an open loop.
    - whstl 3y ago
      Yep. I'm otherwise happy to help with things in the middle of the night, but I need the means to actually diagnose and fix it. As it is, the Ops manager is probably going to have a stroke due to overwork, and it wasn't for lack of people wanting to help.
  - KaiserPro 3y ago
    > The fun part is Ops wants us engineers to help more, but they don't even trust (and some don't know how to run) my Terraform or CloudFormation scripts (I let them choose which), so they just read the tf/yaml and click AWS buttons manually. Often making lots of mistakes. Yeahnah, you're part of the problem. If you've not documented it properly, and more importantly, if the team are weary of prodfucking TF/CF scripts, there is a culture problem. You both need to understand how that works, if they don't have enough time, then you're understaffed, that needs to be fixed. I worked at place that was going through a really painful ops->devops->dev oncall transition. Ops still had a shit ton of power, but nobody included them, and just shouted at them when stuff went red. It took a huge culture shift, and devs sitting in ops and doing a shift or two, to sort it out.
- nunez 3y ago
  That was never supposed to happen, and DevOps enthusiasts still think that's a joke. Unfortunately, at some places, that was the _only_ way to get higher-up people thinking about the culture. Kind of like the idea of a scrum master, which was never supposed to be a role
campbel 3y ago
The creation of DevOps teams instead of SRE / Platform seems pretty common and I think originates from ignorant engineering leaders and poorly defined roles and responsibilities. You want to leverage the expertise of the folks who focus on cloud infrastructure, but you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput and reliability.
- BeefySwain 3y ago
  Can you expand on what you mean by this? Specifically: "you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput and reliability"
  - groestl 3y ago
    I guess it's: You change the way you think about those metrics if you're the one being called at 2AM Saturday night.
  - Etheryte 3y ago
    Your application architecture dictates how it should be deployed and the other way around, if you try to decouple the two you're bound to run into issues. Depending on what your application does, high throughput with some loss might be ideal, or you might need at least once delivery instead while sacrificing throughput, or any other dynamic. If your deployment is done by an entirely different team from the team that builds the product itself you have little hope of predictable positive outcomes without a lot of pain.
  - formercoder 3y ago
    SRE needs authority to hand back the pagers. That’s a key aspect.
  - ddaaeugzzgg444 3y ago
    [dead]
  - 3y ago
smcleod 3y ago
Yep, bang on. As for the PE being the “solution” it is incredibly important and sorely needed but it won’t solve your enterprise culture.
- dopylitty 3y ago
  Regarding enterprise culture I’m curious who will be in this “platform engineering” team and what expertise they’ll have. In (enterprise) reality it will more than likely be a bunch of sysadmins and former “network/storage guys” who saw which way the wind was blowing and got a cloud cert. They really have no business forcing a particular infra pattern on application teams because they have no application level expertise and nothing about adding LUNs to ESXi, typing Cisco configs, or writing bash scripts to automate yum update gives you expertise in application design/architecture. Then because they know nothing about application level concerns the PE team’s golden patterns will either be ignored or will be customized for every application to the point that they’re unmaintainable. Note this is the enterprise view. It’s probably different at tech focused companies.
  - smcleod 3y ago
    I think that once a company is big enough you can have a few options to get code to prod and make sure it stays running as expected. You could have an option where a team is focused on building most of the platform up to an agreed point (e.g. everything under an application’s container image) and provide some common ways of setting up application monitoring / logging etc…. Making sure that while there is a core internal platform as a product team that you’re focused on being a capability rather than a business unit - by that I mean you should have some if not most folks embedded (or rotating through) dev teams. You can also have an option that lets teams self organise how they get their code running - with some light touch guard rails and potentially caveats for applications with PII etc…
  - KaiserPro 3y ago
    > In (enterprise) reality it will more than likely be a bunch of sysadmins and former “network/storage guys” who saw which way the wind was blowing and got a cloud cert. This sounds like a tale of woe. but. The fault is the culture, not the people doing the grunt work. Like devs, if you're used to nails, then you'll want to use a hammer for everything. > gives you expertise in application design/architecture. a sysadmin should know much better how your application runs in real life, and more importantly, how it behaves when it goes to shit. They _should_ know more about all the different types of services (ie kafka vs NATS etc) because they will have had to set way more of them up than a typical dev. If thats not the case, you've had bad or poorly mentored juniors running your stuff. The reason they should know, is because they have the pager to make it work at 4am. Look, if I had it my way, and I was at a medium sized company, I would buy in 3 mainframes, in three disparate locations, set the lock step replication to "very yes" and make sure each dev used the local DB and use the job description lang, to define scale. Pay the $1million a year to outsource reliability of DB and state. Because frankly most people can't program for either scale, redundancy or recoverability. (yes that includes k8s) let the hardware do all of that and save a bunch on staffing costs. I work at a HUGE scale place, and yes there is a need for special sauce there. but even then not that much, and not enough to justify the massive churn in systems. Most PE is just cargo culting busy work. Stick to ECS and PSQL and just get on with making actual buisness features.
expertentipp 3y ago
I'm technologically stuck in the era when everyone hated JavaScript. I provision instances with bash and services over REST APIs.
- nailer 3y ago
  To be fair JS was a bit ridiculous pre ES2017. These days, you have async/await (even at the top level) and it just works when you use `node file.mjs` For TS, install '@digitak/esrun' and run: esrun file.ts (no tsconfig, package.json, etc. required) That basically turns JS into the async equivalent of python and is great for boring sysadmin tasks.
- elyoun 3y ago
  I just started working on a more devops/platform kind of role. I believe we're basically doing what you feel like is stuck in an era. Can you shed some light as to how JavaScript is used/ is an improvement?
- candiddevmike 3y ago
  I have a tool that I'm about to release soon that you may be interested in. It's a way to configure servers (with bash or whatever you want) in a distributed, decentralized way. You build "patterns" which are signed build and run scripts that the servers can be pushed or pull the patterns from a storage provider (like a storage bucket). The tool exports various prometheus metrics that you'd use to determine the state of it and whether the patterns were successfully applied etc. Best of all, it's stateful by design, so when you add or remove things the pattern will clean up after itself. Message me if you want to be part of the beta.
hk1337 3y ago
This seems like the same thing as with “Agile”. The name is used and contorted into some bastardized form so now we just throw the whole thing out.
- hinkley 3y ago
  Semantic dilution.
batmansmk 3y ago
Whats your SRE / DevOps ratio? In my experience, in the 10 teams I had or oversaw, I had about 1 FTE in DevOps for 15 SRE. Of course it depends, but what’s your personal experience?
- smcleod 3y ago
  DevOps isn’t a role, it’s values and ways of working.
  - SerCe 3y ago
    Yes, unfortunately, the misconception that DevOps is a role or person, rather than a culture, is the root of most problems with DevOps, DevSecOps, and DevSecMlTriceratops.
Pannoniae 3y ago
DevOps just makes the problem worse which it is intended to solve. DevOps started from "deploying is too hard, and the developers don't know how to run complicated shell scripts to deploy our software to release, anyone should be able to deploy it and have it scale!". However, most companies started to cargo-cult Google and just built everything in the "most scalable" way, incurring tons of overhead, both in deployment friction and hardware costs, so we are back to DevOps being an almost completely separate field from software engineering, and the programmers still can't deploy software.
- ecshafer 3y ago
  Devops absolutely fixed a lot of issues. Going from manually merged SVN branches on a quarterly release monolith to a "click and release application" is a huge saver in developer time and pain.
  - theduder99 3y ago
    that's great for you. at my company devops are still doing monthly releases and unable to deploy single microservices because all the services are too interconnected.
    - llama052 3y ago
      That doesn't sound like a devops problem, but a culture one.
- beezlewax 3y ago
  We use simple github workflows to release microservice based services and frontends on a daily basis. They're pushed through a series of dev environments for testing before being "released" into production by managers. It's not hard for devs at all.
- donutshop 3y ago
  > DevOps just makes the problem worse which it is intended to solve. DevOps started from "deploying is too hard, and the developers don't know how to run complicated shell scripts to deploy our software to release, anyone should be able to deploy it and have it scale!". However, most companies started to cargo-cult Google and just built everything in the "most scalable" way, incurring tons of overhead, both in deployment friction and hardware costs, so we are back to DevOps being an almost completely separate field from software engineering, and the programmers still can't deploy software. Absolutely this. Instead of collaborating to solve problems we are wasting time on pet projects.
caprock 3y ago
DevOps as a culture vs DevOps as the neo modern unix admin has no right answer. It's just like software development philosophy and organization. It's a fine approach, if you have a well aligned group of people working with the same understanding. The same is true for places that prefer to draw boundaries of responsibility in other ways.
steve1977 3y ago
If you have a DevOps team, your company probably doesn’t understand what DevOps is about.
- dijit 3y ago
  Given that everyone and their mother has slightly nuanced yet disperate definitions of exactly what devops is, I'm not sure. DevOps? Oh you mean the developer experience people? DevOps? Oh, the build pipeline people? DevOps? Oh, the people who handle the cloud? DevOps? Ah, you mean those folks who handle our observability! DevOps? That's Katya, she's a developer but i'm not sure what she does except act stressed out all the time and slow us down.
  - steve1977 3y ago
    Hint, in my definition it’s none of the above
pizzafeelsright 3y ago
All this IT stuff fails, needs to be replaced, repaired, or adapted to a new situation. The best solutions, whatever the methodology and avenue of implementation, are made better when the builders "live below the dam." The incentive to build the best dam is when the party responsible must live below the dam. The best metric is customer retention. Directors of customer support need to be on the phone with escalations, not just watching metrics. There's a leak? Take the complaints. Embrace the hate. Fix the damn dam.
politelemon 3y ago
If the DevOps "culture shift" didn't solve problems for you, this will not either. You will still have similar problems, now abstracted away into a different team that operate at their own pace. They will also be overbooked, like before, with the difference that now they are serving multiple teams you didn't have to contend with before. They will not share your incentives or priorities for specific features. Their time will still not be taken into account during product planning. Both the DevOps move, and its current issues, as well as platform engineering are organisational problems. Your organisation needs to be set up around this in order for it to succeed. Notice that the article paints platform engineering with the same utopia like brush strokes that once painted "DevOps as a culture", only to be mangled in its handling and blamed for the problems that it brought about.
dmart 3y ago
To me, DevOps culture just feels like a way for businesses to save money by not staffing a dedicated infrastructure team and pushing all those responsibilities onto application developers. The amount of time I've spent fiddling with Terraform, Ansible, Kubernetes manifests, Helm charts, Jenkins configuration, GitHub Actions configuration, AWS IAM, and so on over the past few years is absurd, probably more than the time I've spent writing actual application code.
- booi 3y ago
  What now? If you’re spending that much time on devops I feel like you’re doing it wrong. I do eng (team of 7) and devops (only me) and I feel like I spend about 5% of my time on devops. These tools have made deploying production quality infrastructure take a fraction of the time. These environments are basically already preaudited and verified. They’re also just much higher quality in general. It’s quite typical for different projects to share infrastructure in traditionally deployed systems but with devops we can have much better isolation with less deployment time.
- satvikpendem 3y ago
  We have separate devops teams, though, so I do zero fiddling with all those technologies.
- pdimitar 3y ago
  > To me, DevOps culture just feels like a way for businesses to save money by not staffing a dedicated infrastructure team and pushing all those responsibilities onto application developers. Yeah, because that's exactly what it is.
- report-to-trees 3y ago
  One of the things that bothers me the most with the cooperate software development rat race is how many problems are being solved over and over again. Every company is staffing their own devops teams to build their own abstractions over these technologies so app developers don't have to worry about it. I personally know multiple devs who basically move from company to company reimplementing the same devops tools at each one. It all just feels like a collosal waste of energy and collective resources.
pojzon 3y ago
DevOps is terrible because we want to have a team of fullstack expert engineers that have certs in every tech, but pay them for a single role. First there are not enough ppl with that huge landscape of knowledge. Second you would have to pay a salary of 5 ppl i.e. 500k/y to make ppl willing to spend that much time on work.
- davewritescode 3y ago
  There are more people than you think that can slap together some YAML and a Helm chart and deploy something reasonably resilient inside a Kubernetes cluster. I've been around for the cloud migrations in every phase, what we have today with Kubernetes is miles better than the maze of Puppet and other crap we had 20 years ago.
  - pojzon 3y ago
    But thats just a tip of an iceberg. What happens when your Cillium Service Mesh starts misbehaving due to missed network packets ? Or when you have to setup a resillient backup of a cluster that runs 30 000 pods on 300 nodes ? And upgrade. Or when you have to provide a telemetry for highly distributed system using 5 languages on landscape spread over frontend and backend with three different db engines and two clouds ? Kubernetes is easier to start but 10 times as hard to do at scale you didnt even have back then in OnPremise era. Yes you are right there are a lot of ppl who can write and deploy „Hello World” apps, but thats about it.
winternett 3y ago
Since the in house server days, technology solutions have only become more and more time consuming to implement, ID & PII obsessed, and bulky (data footprint-wise). I am thankful I remember much more simple times, when apps were tiny, run on regular computers in my office, and you could symply restore from a local backup to bring a service back online. Now we have companies paying thousands of dollars a month to host a simple site or web app, because now it's all in the cloud, no matter the apps function, while hosting customers assume about 80-90 percent of responsibility for securing and maintaining their app anyway. Containerization is leveraged to add a "backdoor" for individual admins, but that only serves to protect infrastructure, it actually makes a customer's individual app open to vulnerability that same as running your own instance if you miss vital updates or if there is a breakdown in the supply chain. We need to stop listening to the companies that market tools and be honest with ourselves... Security needs a better model than just adding new tools and entry points to the app chain. There's gotta be a point where you evaluate things and tell yourself that there has to be a better and more affordable way than tunneling through 5 VPNs just to push updates for 8 disparate JS and PHP libraries and 5 containers. App and library updates are also far too frequent in 2023 as well... Frameworks need to ship instances with less features, and modules should be reduced to only those used and deemed most essential, reducing footprint is a key aspect lost on technological advancement, it is also a firm indicator of increased efficiency for app and library devs if you ask me. Now with zero trust, devs literally spend a lot of their daily workload logging back in and re-establishing VPN connections due to timeouts. There's got to be a point where we develop a far more simple IT solution to all of this mess. Security is still getting compromised regularly no matter what is done. The solution also won't likely involve Ai at this point in my opinion, as security and complexity are assuredly not resolved by leveraging current-state Ai tools. It's also important to note this is why PHP is still going strong, it doesn't require compilation, and is relatively easier than most other langs on learning curve to implement. I literally built my IT career on making things simpler, and in explaining complex IT issues in human language to non-technical people. There is a lot of room in my field for growth due to the rest of the industry's constant focus on jargon and over-complexity. Simpler solutions, conveyed and implemented in human language, will be king in 2024.
- xnx 3y ago
ChoHag 3y ago
[dead]
KaiserPro 3y ago
I think the biggest issue with devops is that it originally meant "socialising" your sysadmins, by getting them to sit with your devs, so that shit didn't get lost because nobody thought to talk to the right team. But then it morphed into "oh lets innovate with infrastructure" but the innovation turned into "lol lets just restart from scratch and ignore history" Anybody who used early k8s can attest to how un production ready it was for the longest time. It _felt_ like progress because it required a lot of (to dev eyes) hidden magic to make it work. Now, we are back with sysadmins, but they are called SREs. They provide a platform, which devs talk to to deploy stuff. Basically we're back where we were in 2015, just with more yaml. The good things that have come out of devops has been the dashboarding and metrics tools. Its just a shame that everything else touched by it appears to be an infinite source of busy work (looking at you kubernetes) (before anyone asks, I'm an SRE at a FAANG.)
- notanormalnerd 3y ago
  So you mean, DevOps has made Sysadmins more service oriented and now instead of having their pets and being in overprotective silos, they actually provide best practices and proper platforms for the poeple developing the software?
  - KaiserPro 3y ago
    thats how it _should_ be. The place it worked the best was where a team of sysadmins were split up and embedded with each dev team. each devop would then rotate onto another team every 6 months or so. This meant that _we_ had to document our shit for the next devop, but also helped eliminate key person dependencies. We would then have a weekly offsite where we'd bitch and moan about our teams and conspire to make things better.
  - sisve 3y ago
    Im not sure how former sysadmins feel about it, but from a dev standpoint, DevOps worked great for us. Making a clean cut on a plattform. Was my app down, I got message/called. If the platform was down it was the Infrastructure team / PaaS provider that needed to fix it. In the old days we had so few deploys and a hotfix was something that was extraordinary. I actually had a board of something in another country approving that we where allowed to deploy. They knew nothing about the application..
mschuster91 3y ago
Yuck, Portainer... when I read that I get nightmares. The problem is: when you give developers access to Kubernetes, to ECS, to Portainer, to whatever they. will. not. care. about literally anything. You'll find a hotpot of cobbled together Dockerfiles with zero provenance, with base images pinned to stuff years old, and completely inefficient layer ordering. Pipelines won't use caching or parallelism (okay, sometimes because they use Maven which is a clusterfuck on its own). Software developers are developers, 90%+ will never have heard of how Linux works below the hood, how Docker actually works, they'll just copy and paste together stuff from Stackoverflow without even attempting to understand how it works. And I don't even blame them. I can't, I don't want and I won't. Companies, it's time to stop loading stuff onto your developers that they don't have to (have) any clue. JFC.
- ElectricalUnion 3y ago
  The developers are literally paid to do the least amount of work that could possibly do towards delivering some abstract MVP "this sprint". Actually making something that is simple and maintainable is the 7th or 8th priority of business, if it is a priority at all. Artificial "obstacles" to delivering the artifacts in time in fact, wanted, a sort of corporation Deus ex machina, externalities to explain why they did not meet delivery deadlines last time. Incentives are not aligned. Things will remain broken until incentives are aligned.
- shrimp_emoji 3y ago
  And some devs do learn how Linux works but don't want to waste their time memorizing how shitty $DEVOPS_PRODUCT that'll be replaced in two years works. :D
  - mschuster91 3y ago
    Indeed. Which is why I waited for three years to take on Docker, and two more for Kubernetes after I got burned with DC/OS, and why I entirely left the NodeJS/frontend ecosystem. I'm too old to act as the ripener for bananaware any more, and I'm pretty sick of companies exploiting other companies or, even worse, individual developers to act as their free QA department.
lifeisstillgood 3y ago
Nobody knows anything The difficult paradoxes have not been resolved No-one can say this loudly because there is no safe space
- lifeisstillgood 3y ago
  Nobody Knows Anything For a given level of complexity in an organisation, talk to any 10 experts on the subject in the organisation (ie how does the foobar process work) and you will find blind men describing an elephant. All technically correct but all missing important parts The paradoxes At some point an organisation acquired or grew an antogonistic part. Both cannot really exist but both have not been removed. Think either Google search vs Google ad sales (if you return perfect search terms each time, you get less page views hence less ad impressions. Make search worse get more ads served). Or simply other non core businesses. Pretty much every non-tech mega corp is really dozens and dozens of smaller businesses with a shared treasury so they can ride out each others ups and downs. But one or two businesses are really carrying oat of the load. Anyway, you can find paradoxes or conflicts of interest in all businesses and badly aligned incentives hurt everyone. Someone should fix that. Not me I don't have the power (indeed usually only the orignal centralmfounder has power to) Safe spaces Yeah these don't exist. If you ain't upbeat publically you are seen as trying to pull everyone else under. So how are problems acknowledged let alone fixed? Dunno. What would really help would be free speech, open conversations and discussions based on data that can inform policy and provide impetus to chnage. we (kinda) have this in western journalism / media (including social media). But absolutely no large corporation anywhere has a open discussion area where different factions can raise their concerns. there are of course political infighting amoungst the powerful elites of the corporation but that's not the same thing - they might even be accurate about the problems but no one else knows. Democracy matters Sorry this was supposed to be about devops - and it kinda is. Most large corps have some devops thing going on where "this time we will fix it all for everyone" - but has there been open conversations with data supporting it from all parts of the org? oh come on.
k8svet 3y ago
"DevOps" as in Developers being closer to Ops, or "DevOps" as in "I'm a shiny Golang outfit that is going to solve Real Problems TM with YAML and a bunch of templated-generated Golang?" I realize this is a bit of an odd tangent, but I would love to hear cadey talk more about what "devops" means at a shop like Tailscale. Actually, as I type this, even more curious. I know some of those folks are cut-me-and-I-bleed-Go types, but I also know they're deep into NixOS for server management. It would be very interesting to hear more about how those intersect for their infra/server/dev culture.
dilyevsky 3y ago
Most commenters here seem to ignore the fact that saas deployments have become increasingly complex over last decade in part bc low hanging fruit has been picked. Also the massive scale of even relatively no-name companies out there. I'd add that most of the 1-4 "problems" (boo-hoo, managing IAM is boring for devs) OP mentioned as well as advent of so called "platform engineering" stem from one thing only - poor engineering leadership.
moron4hire 3y ago
What drives me nuts is the number of places that expect you to have deep experience with whatever CI/CD solution they're using before they'll even consider you for a position. How have we gotten to the point where deploying an application is so fucking arcane of a process that you need to have 5 years of experience with whatever acronym soup some CTO bodged together or you are considered worthless?
pdimitar 3y ago
Replacing "DevOps" with "Platform Engineering" is not waving a magic wand and removing problems. The original article's main point is: "use DevOps marketplaces", more or less, but that comes with its own set of headaches (mainly auditing and making sure you trust them, which might easily take more time than rolling your own stuff). I fully stand behind the "just have programmers that write internal tooling" recommendation though. Over the course of my long career this is what has worked best almost every time, and these are the programmers who get the least amount of respect even though they have to dance and juggle on a burning rope more often than not.
fmajid 3y ago
Someone at a Linux company once said: if you combine "Terraform" and "Ansible", you get "Terrible".
jiggawatts 3y ago
In my opinion, more than half the problem is the tooling, as indicated by the cartoon in the linked article. DevOps pipeline languages are generally either weakly typed or stringly-typed. They generally expect users to do the control character escaping, in several different obscure formats. Did I say escaping? I meant three layers of nested escaping! Variable/macro substitution as a rule uses syntax that overlaps with either PowerShell, Bash, or both because otherwise it would be too easy. Another basic thing is the absence of a debugger — I have to run a complete pipeline with a change to dump out environment variables! Instructions contain the display names of paths, scripts use the variable names, but the output (and errors!) use the values with no clear mapping backwards. What the eff is C:\s\1\x? I dunno, but it caused an error! The worst sin is not having a local development experience where the entire pipeline can be run without having to commit code and push stuff almost all the way to production. Even if the pipeline agent is made available, it’s useless without the underlying VM image which is huge and not easily obtained. Not to mention KMS/KeyVault and other similar services on which pipelines can depend. Developers complain about having to do DevOps, but in my opinion the problem isn’t who is doing it, but how. The tooling is just so bad, so very very bad, that anyone would hate it if they were the ones forced to do it. I’m an ex-dev-now-SRE and I hate it.
cdnsteve 3y ago
Isn't this what PaaS is suppose to solve?
aloknnikhil 3y ago
This is the original author / article - https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88162c86d7 https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88...
hgopolis 3y ago
Sysadmin, devops, platform eng all suffer from made up problems. What’s IAM? Access control. What’s a firewall? Access control. What’s a route table? Access control. What are resource limits and reservations? Access control. Yet years of legacy solutions has us discussing all these things as if they are fundamentally different. User management comes wrapped in vacuous jargon. Network addressing schemes are completely design by the pocket protector committee (fuck off they can be wrapped in abstraction that is less obtuse jargon and more memorable/mnemonic). There are no users, groups, or network routes in the machine. Devops is obtuse access control because a lot of companies can make money from insuring its obtuse access control.
- rswail 3y ago
  Calling a route table "access control" is not useful. It's not access control, it's literally in the name, it is about navigation of the network via routes. As for your comment about the "pocket protector committee" that's just ad hominem attacks on the people that keep the internet running. Saying that NDP is more complicated than ARP/RARP etc because it has a different name is, again, attacking actual needs because it's more complicated than the protocol it replaces. A network ACL is access control (it's in the name), L2 VLANs may be used for network separation, but they're also used for switches to be able to not have to broadcast packets across all their ports. If you want to downplay the different requirements that the different forms of resource management require as "access control", then all you're doing is removing all meaning from the term "access control" in the first place.
  - hgopolis 3y ago
    I design motherboards for network devices; I keep the internet going. Routes are inaccessible without access to them. The underlying machine states to check for access to a route is algorithmically similar to the other checking for user access. It’s unfortunate so many are allowed to work in IT with cliff notes level awareness of how technology works. The layers of indirection you work with are the knobs and buttons people like me choose. You merely obfuscate with vacuous jargon to serve your career goals.
skyrocketer77 3y ago
The original article is here, this guy just copied it, lol https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88162c86d7 https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88...
- donutshop 3y ago
  Underrated comment
high_5 3y ago
Devops got stunted or intermingled with the great migration to the Cloud. It could be practiced on any level of infrastructure, but the discourse currents of the Cloud were too strong.
avryhof 3y ago
Where I work, it just means I have to maintain my own infrastructure in addition to being a developer.
nunez 3y ago
2023 DevOps is awesome. Today's "sysadmins" are kind-of, sort-of expected to know a programming language, at least superficially. This was definitely not the case in 2015. The whole idea of Platform Engineering formally existing (I.e. platform is a product) is fantastic. There has always been lip service from IT about treating the rest of the business as customers, but the definition of "customer" was usually bastardized to fit whatever agenda IT execs wanted to push. Not the case (as much) anymore. Also, the idea that today's "sysadmins" are part of application delivery into production was extremely nascent in 2015. Most sysadmin jobs didn't touch apps with a 10-ft pole back then. This, IMO, is the greatest benefit the movement brought to bear. Shoot, even the fact that saying DevOps feels dated because it's obvious now demonstrates how far we've come. Saved the best for last. Kubernetes is hella complicated compared to 2015 "DevOps" tech stack (which i miss because they are simple), but provides so so so SO many benefits for free, especially if you're not in public cloud.
- dijit 3y ago
  > Today's "sysadmins" are kind-of, sort-of expected to know a programming language, at least superficially. This was definitely not the case in 2015. There was a "weird period" in 2010-2014 where mid level sysadmins didn't know much scripting or automation. I'm not sure how that happened. Senior sysadmins always scripted, Ruby was the de-facto choice for those folks in 2010-2015; before that it was mostly perl, and after that it seems to be mostly python. Mid-level sysadmins definitely knew bash and perl in 2000-2010. I think it was the rise of calling level 3 helpdesk "sysadmin" that sullied the brand. But the same will happen with other disciplines. I'm sure I'll hear in a couple years about SRE's or DevOps who can't code. Heck, I bet there are already devops who can't code, who only know how to template yaml.
donutshop 3y ago
Looks like OP copied and pasted the entire article from somewhere else. https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88162c86d7 https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88...

116 points by moonabid786 3y ago

jlbooker 3y ago

This return a 404 now. Anyone have a better link?

MostlyAmiable 3y ago
Apparently it was plagiarized from here: https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88162c86d7 https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88...

hinkley 3y ago

The fact that we have a “devops team” at work infuriates me. It’s enough to distract me me from how “Scrum” has destroyed Agile.

0x008 3y ago
What is so horrible about it? Our devs are happy they can focus on fixing unit tests in pycharm, while others care about where the code is going.
- pjmlp 3y ago
  Because that is the software developer/systems engineer division that already existed 20 years ago. The idea behind devops was to be a single team.
  - bojo 3y ago
    > The idea behind devops was to be a single team. I assume you are pointing out that split-teams is the insane part, in which case I agree. The philosophy was supposed to define the blurred line between developers and operations, using tooling to help facilitate the shared responsibilities where sensible and make hand-off a breeze. As you pointed out, the division existed 20 years ago, and it still exists now at many places. That lack of integration doesn't offer much improvement in relation to the past.
    - ddaaeugzzgg444 3y ago
      [dead]
- matsemann 3y ago
  Because it's just a hip way of saying you have a sysadmin team. How can something or someone be "devops" if all the developers are in one team doing dev tasks, and the ops in a separate team doing ops stuff?
  - dijit 3y ago
    If we keep reinventing the same paradigms; isn't it a perfectly fine paradigm? It's unfortunately inherent that some people want predictable, scalable, observable, stable- and at the same time they want fast moving and many new features. I think it's not so much that dev and admin work is terribly difficult, it's more that "a man cannot serve two masters" and someone will be focusing on stability and someone else will be focusing on quickly deploying. Trying to do both yourself I think leads to burnout.
  - root_axis 3y ago
    The term is meant to encompass the "infrastructure as code" workflow. The "dev" in "devops" refers to the skills necessary to read and write code that manages the state of the system infrastructure.
  - ecshafer 3y ago
    A devops team is in practice typically very different than a sysadmin team. Good sysadmins automate the hell out of everything, but it’s usually not something other people can use. Devops teams automate all of that stuff but then they also let teams build new infrastructure when they do deploys and automate that whole process, but it’s client facing. It’s development and operations, it’s a pretty good name, though I think platform or production engineering is typically a better name.
    - hinkley 3y ago
      The quality of devops scripts, from the perspective of a person who likes to write code that has to work right the first time, is about two standard deviations better than sysadmin scripts. But still of a par that would keep someone from getting promoted past a Senior Dev title. What’s worse is they can be condescending while shoveling out absolute garbage. If I’m honest that’s the part that bugs me the most.
- medellin 3y ago
  The thing i have come to dislike is that in both companies i work for it lead to strange working relationships between ops/eng. anything that breaks in the deploy pipeline is now “the devops team problem” and they normally act like it’s also your fault. In the past this has meant people messaging me saying they are blocked because their dockerfile fails in CI. Then you walk through it with them and it’s also broken locally and it comes down to them pinning to something like latest tag and the image “shocker” changed. But now you are fixing it and after you go remind everyone that you should pin your image to as strict as possible image tag. But no one cares because they can move fast and break things and when it does blame it on CI and now it’s not their problem. Anyway this is just one tiny example but the same underlying thing starts happening in all parts of the development cycle. “My deploy doesn’t work” … well your program has an error and crashes on startup but looking at the logs would be too much work. Let’s just say we are blocked and log off. As you can see this is a culture issue but it seems to be common when you have a devops team. And even more common when management (read senior management and up) think that developers “should just be developing”. Anyway thats my rant.
  - anotherhue 3y ago
    I have seen infra/devops scapegoated too much. Half the time the developers in question won't even read the error output before raising a ticket. The only solution I see is to enforce commit checks before hitting shared environments, but that comes with its own set of complaints.
    - tstrimple 3y ago
      > Half the time the developers in question won't even read the error output before raising a ticket. If they had no DevOps team to raise a ticket with, and it had to get done they would find some way to figure it out if they had the ability to modify and initiate their own pipelines. The reason I like "you build it you run it" is I feel like I have far more control over my own destiny. You're not having to negotiate and coordinate with a separate team through some ticketing system. Far less wasted back and forth and miscommunication. If that means you have to expand one of your teams with an additional DevOps person to foster understanding of the tooling on a team who hasn't done it before that's fine. But don't make them the DevOps person. Their goal is to enable the rest of the team to pick up these skills. Their goal should be to transition into a normal contributing member of the team working on product development because the team no longer has a need for a DevOps person.
- steve1977 3y ago
  That’s fine, it just has nothing to do with DevOps then. The whole point is that these are not separate teams.
- Twirrim 3y ago
  So in other words, you made silos. More throwing things over the wall and having it be someone else's problem. That's fundamentally the antithesis of DevOps. People do not fix the pain they don't see, and managers doubly so. By shifting operations off to a "devops" team, you just end up with developer teams that don't know about, and don't care to fix, the things that make it painful to deploy or run their code in production. Those that have to deal with the pain rarely have any political leverage they need to even enact change. There's zero motivation for either the developers writing the software _or their management_ to fix it. Dev manager incentives are purely aligned with getting code and features built. To give a nice real world example, building out new regions for AWS services used to be a pain in the arse (since I left I hear some major top down initiatives have fixed that). The service team I worked for, almost doubly so painful for region build. Standing up an entire region was something like ~50 days of engineering time. Not because it was especially hard, per se, but because the ops team always had to wade through piles of refactored shit that introduced new circular dependencies, or wasn't documented, or broke other aspects of any automation we could produce. (I remember one DynamoDB schema overhaul, where we did a complicated migration. Next region build, service components wouldn't stand up because we'd used the automation to create the new table schemas that had replaced the old table schema entries, but the service code still expected to see the old schemas in the right format. It took several months to get the dev manager to agree to engineers spending time cleaning up old schema references from their code base.) The dev manager incentives were not aligned with making it easy. It wasn't their engineering resources being consumed, and they had features they needed to ship. The developers never changed the way they worked because even though they were told the issues (and even though they were, virtually to a person, great people to work with, considerate etc.), they never experienced the pain so they never thought about those consequences of what they were doing when they were doing it. Ops manager never had any leverage, because every single time we managed to get a region launched in time as to not be an issue, and every time things were down to the wire it was clearly as a result of the ops team not building the right automation :eyeroll:, rather than that the dev teams were making some kind of brand new nightmare each time. It wasn't until the ops team had lost enough members of staff to that BS, and region build shifted on to the developers themselves, that suddenly most of the problems went away, never to appear again. They were aware and conscious of the pain certain decisions would cause because they had felt it themselves, and they made sure not to introduce more pain. DevOps is about making sure incentives and politics align to make things overall better. Shoving it in to a silo is literally the complete opposite.
- stavros 3y ago
  It's like saying you have a self-service waiter.
tstrimple 3y ago
I think there is room for a DevOps team if they are focused on creating and maintaining the overall DevOps platform. But if they are taking tickets and setting up pipelines for individual applications, they are doing it wrong. But a team focused on making a streamlined platform with ready to adopt patterns that already account for compliance and security related concerns, they can spread DevOps adoption more effectively and consistently throughout the organization.
whstl 3y ago
The only thing I like about the return of sysadmins and the end of the "you build, you run it" culture of Devops it is that I'm 100% off the hook if something dies in the middle of the night. The so-called DevOps guys are stressed out of their marbles, but when the CTO calls me 4 AM I can't say anything but "Sorry dude, I'm not a clairvoyant. I have no permission to do shit in prod, call Ops or give me AWS access now". He stopped calling. The fun part is Ops wants us engineers to help more, but they don't even trust (and some don't know how to run) my Terraform or CloudFormation scripts (I let them choose which), so they just read the tf/yaml and click AWS buttons manually. Often making lots of mistakes.
- hinkley 3y ago
  I always do a better job of writing logic when I sympathize with the victims. Some of those late night calls are devs setting ops up for failure. It was meant to close an open loop.
  - whstl 3y ago
    Yep. I'm otherwise happy to help with things in the middle of the night, but I need the means to actually diagnose and fix it. As it is, the Ops manager is probably going to have a stroke due to overwork, and it wasn't for lack of people wanting to help.
- KaiserPro 3y ago
  > The fun part is Ops wants us engineers to help more, but they don't even trust (and some don't know how to run) my Terraform or CloudFormation scripts (I let them choose which), so they just read the tf/yaml and click AWS buttons manually. Often making lots of mistakes. Yeahnah, you're part of the problem. If you've not documented it properly, and more importantly, if the team are weary of prodfucking TF/CF scripts, there is a culture problem. You both need to understand how that works, if they don't have enough time, then you're understaffed, that needs to be fixed. I worked at place that was going through a really painful ops->devops->dev oncall transition. Ops still had a shit ton of power, but nobody included them, and just shouted at them when stuff went red. It took a huge culture shift, and devs sitting in ops and doing a shift or two, to sort it out.
nunez 3y ago
That was never supposed to happen, and DevOps enthusiasts still think that's a joke. Unfortunately, at some places, that was the _only_ way to get higher-up people thinking about the culture. Kind of like the idea of a scrum master, which was never supposed to be a role

campbel 3y ago

The creation of DevOps teams instead of SRE / Platform seems pretty common and I think originates from ignorant engineering leaders and poorly defined roles and responsibilities. You want to leverage the expertise of the folks who focus on cloud infrastructure, but you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput and reliability.

BeefySwain 3y ago
Can you expand on what you mean by this? Specifically: "you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput and reliability"
- groestl 3y ago
  I guess it's: You change the way you think about those metrics if you're the one being called at 2AM Saturday night.
- Etheryte 3y ago
  Your application architecture dictates how it should be deployed and the other way around, if you try to decouple the two you're bound to run into issues. Depending on what your application does, high throughput with some loss might be ideal, or you might need at least once delivery instead while sacrificing throughput, or any other dynamic. If your deployment is done by an entirely different team from the team that builds the product itself you have little hope of predictable positive outcomes without a lot of pain.
- formercoder 3y ago
  SRE needs authority to hand back the pagers. That’s a key aspect.
- ddaaeugzzgg444 3y ago
  [dead]
- 3y ago

smcleod 3y ago

Yep, bang on. As for the PE being the “solution” it is incredibly important and sorely needed but it won’t solve your enterprise culture.

dopylitty 3y ago
Regarding enterprise culture I’m curious who will be in this “platform engineering” team and what expertise they’ll have. In (enterprise) reality it will more than likely be a bunch of sysadmins and former “network/storage guys” who saw which way the wind was blowing and got a cloud cert. They really have no business forcing a particular infra pattern on application teams because they have no application level expertise and nothing about adding LUNs to ESXi, typing Cisco configs, or writing bash scripts to automate yum update gives you expertise in application design/architecture. Then because they know nothing about application level concerns the PE team’s golden patterns will either be ignored or will be customized for every application to the point that they’re unmaintainable. Note this is the enterprise view. It’s probably different at tech focused companies.
- smcleod 3y ago
  I think that once a company is big enough you can have a few options to get code to prod and make sure it stays running as expected. You could have an option where a team is focused on building most of the platform up to an agreed point (e.g. everything under an application’s container image) and provide some common ways of setting up application monitoring / logging etc…. Making sure that while there is a core internal platform as a product team that you’re focused on being a capability rather than a business unit - by that I mean you should have some if not most folks embedded (or rotating through) dev teams. You can also have an option that lets teams self organise how they get their code running - with some light touch guard rails and potentially caveats for applications with PII etc…
- KaiserPro 3y ago
  > In (enterprise) reality it will more than likely be a bunch of sysadmins and former “network/storage guys” who saw which way the wind was blowing and got a cloud cert. This sounds like a tale of woe. but. The fault is the culture, not the people doing the grunt work. Like devs, if you're used to nails, then you'll want to use a hammer for everything. > gives you expertise in application design/architecture. a sysadmin should know much better how your application runs in real life, and more importantly, how it behaves when it goes to shit. They _should_ know more about all the different types of services (ie kafka vs NATS etc) because they will have had to set way more of them up than a typical dev. If thats not the case, you've had bad or poorly mentored juniors running your stuff. The reason they should know, is because they have the pager to make it work at 4am. Look, if I had it my way, and I was at a medium sized company, I would buy in 3 mainframes, in three disparate locations, set the lock step replication to "very yes" and make sure each dev used the local DB and use the job description lang, to define scale. Pay the $1million a year to outsource reliability of DB and state. Because frankly most people can't program for either scale, redundancy or recoverability. (yes that includes k8s) let the hardware do all of that and save a bunch on staffing costs. I work at a HUGE scale place, and yes there is a need for special sauce there. but even then not that much, and not enough to justify the massive churn in systems. Most PE is just cargo culting busy work. Stick to ECS and PSQL and just get on with making actual buisness features.

expertentipp 3y ago

I'm technologically stuck in the era when everyone hated JavaScript. I provision instances with bash and services over REST APIs.

nailer 3y ago
To be fair JS was a bit ridiculous pre ES2017. These days, you have async/await (even at the top level) and it just works when you use `node file.mjs` For TS, install '@digitak/esrun' and run: esrun file.ts (no tsconfig, package.json, etc. required) That basically turns JS into the async equivalent of python and is great for boring sysadmin tasks.
elyoun 3y ago
I just started working on a more devops/platform kind of role. I believe we're basically doing what you feel like is stuck in an era. Can you shed some light as to how JavaScript is used/ is an improvement?
candiddevmike 3y ago
I have a tool that I'm about to release soon that you may be interested in. It's a way to configure servers (with bash or whatever you want) in a distributed, decentralized way. You build "patterns" which are signed build and run scripts that the servers can be pushed or pull the patterns from a storage provider (like a storage bucket). The tool exports various prometheus metrics that you'd use to determine the state of it and whether the patterns were successfully applied etc. Best of all, it's stateful by design, so when you add or remove things the pattern will clean up after itself. Message me if you want to be part of the beta.

hk1337 3y ago

This seems like the same thing as with “Agile”. The name is used and contorted into some bastardized form so now we just throw the whole thing out.

hinkley 3y ago
Semantic dilution.

batmansmk 3y ago

Whats your SRE / DevOps ratio? In my experience, in the 10 teams I had or oversaw, I had about 1 FTE in DevOps for 15 SRE. Of course it depends, but what’s your personal experience?

smcleod 3y ago
DevOps isn’t a role, it’s values and ways of working.
- SerCe 3y ago
  Yes, unfortunately, the misconception that DevOps is a role or person, rather than a culture, is the root of most problems with DevOps, DevSecOps, and DevSecMlTriceratops.

Pannoniae 3y ago

DevOps just makes the problem worse which it is intended to solve. DevOps started from "deploying is too hard, and the developers don't know how to run complicated shell scripts to deploy our software to release, anyone should be able to deploy it and have it scale!". However, most companies started to cargo-cult Google and just built everything in the "most scalable" way, incurring tons of overhead, both in deployment friction and hardware costs, so we are back to DevOps being an almost completely separate field from software engineering, and the programmers still can't deploy software.

ecshafer 3y ago
Devops absolutely fixed a lot of issues. Going from manually merged SVN branches on a quarterly release monolith to a "click and release application" is a huge saver in developer time and pain.
- theduder99 3y ago
  that's great for you. at my company devops are still doing monthly releases and unable to deploy single microservices because all the services are too interconnected.
  - llama052 3y ago
    That doesn't sound like a devops problem, but a culture one.
beezlewax 3y ago
We use simple github workflows to release microservice based services and frontends on a daily basis. They're pushed through a series of dev environments for testing before being "released" into production by managers. It's not hard for devs at all.
donutshop 3y ago
> DevOps just makes the problem worse which it is intended to solve. DevOps started from "deploying is too hard, and the developers don't know how to run complicated shell scripts to deploy our software to release, anyone should be able to deploy it and have it scale!". However, most companies started to cargo-cult Google and just built everything in the "most scalable" way, incurring tons of overhead, both in deployment friction and hardware costs, so we are back to DevOps being an almost completely separate field from software engineering, and the programmers still can't deploy software. Absolutely this. Instead of collaborating to solve problems we are wasting time on pet projects.

caprock 3y ago

DevOps as a culture vs DevOps as the neo modern unix admin has no right answer. It's just like software development philosophy and organization. It's a fine approach, if you have a well aligned group of people working with the same understanding. The same is true for places that prefer to draw boundaries of responsibility in other ways.

steve1977 3y ago

If you have a DevOps team, your company probably doesn’t understand what DevOps is about.

dijit 3y ago
Given that everyone and their mother has slightly nuanced yet disperate definitions of exactly what devops is, I'm not sure. DevOps? Oh you mean the developer experience people? DevOps? Oh, the build pipeline people? DevOps? Oh, the people who handle the cloud? DevOps? Ah, you mean those folks who handle our observability! DevOps? That's Katya, she's a developer but i'm not sure what she does except act stressed out all the time and slow us down.
- steve1977 3y ago
  Hint, in my definition it’s none of the above

pizzafeelsright 3y ago

All this IT stuff fails, needs to be replaced, repaired, or adapted to a new situation. The best solutions, whatever the methodology and avenue of implementation, are made better when the builders "live below the dam." The incentive to build the best dam is when the party responsible must live below the dam. The best metric is customer retention. Directors of customer support need to be on the phone with escalations, not just watching metrics. There's a leak? Take the complaints. Embrace the hate. Fix the damn dam.

politelemon 3y ago

If the DevOps "culture shift" didn't solve problems for you, this will not either. You will still have similar problems, now abstracted away into a different team that operate at their own pace. They will also be overbooked, like before, with the difference that now they are serving multiple teams you didn't have to contend with before. They will not share your incentives or priorities for specific features. Their time will still not be taken into account during product planning. Both the DevOps move, and its current issues, as well as platform engineering are organisational problems. Your organisation needs to be set up around this in order for it to succeed. Notice that the article paints platform engineering with the same utopia like brush strokes that once painted "DevOps as a culture", only to be mangled in its handling and blamed for the problems that it brought about.

dmart 3y ago

To me, DevOps culture just feels like a way for businesses to save money by not staffing a dedicated infrastructure team and pushing all those responsibilities onto application developers. The amount of time I've spent fiddling with Terraform, Ansible, Kubernetes manifests, Helm charts, Jenkins configuration, GitHub Actions configuration, AWS IAM, and so on over the past few years is absurd, probably more than the time I've spent writing actual application code.

booi 3y ago
What now? If you’re spending that much time on devops I feel like you’re doing it wrong. I do eng (team of 7) and devops (only me) and I feel like I spend about 5% of my time on devops. These tools have made deploying production quality infrastructure take a fraction of the time. These environments are basically already preaudited and verified. They’re also just much higher quality in general. It’s quite typical for different projects to share infrastructure in traditionally deployed systems but with devops we can have much better isolation with less deployment time.
satvikpendem 3y ago
We have separate devops teams, though, so I do zero fiddling with all those technologies.
pdimitar 3y ago
> To me, DevOps culture just feels like a way for businesses to save money by not staffing a dedicated infrastructure team and pushing all those responsibilities onto application developers. Yeah, because that's exactly what it is.
report-to-trees 3y ago
One of the things that bothers me the most with the cooperate software development rat race is how many problems are being solved over and over again. Every company is staffing their own devops teams to build their own abstractions over these technologies so app developers don't have to worry about it. I personally know multiple devs who basically move from company to company reimplementing the same devops tools at each one. It all just feels like a collosal waste of energy and collective resources.

pojzon 3y ago

DevOps is terrible because we want to have a team of fullstack expert engineers that have certs in every tech, but pay them for a single role. First there are not enough ppl with that huge landscape of knowledge. Second you would have to pay a salary of 5 ppl i.e. 500k/y to make ppl willing to spend that much time on work.

davewritescode 3y ago
There are more people than you think that can slap together some YAML and a Helm chart and deploy something reasonably resilient inside a Kubernetes cluster. I've been around for the cloud migrations in every phase, what we have today with Kubernetes is miles better than the maze of Puppet and other crap we had 20 years ago.
- pojzon 3y ago
  But thats just a tip of an iceberg. What happens when your Cillium Service Mesh starts misbehaving due to missed network packets ? Or when you have to setup a resillient backup of a cluster that runs 30 000 pods on 300 nodes ? And upgrade. Or when you have to provide a telemetry for highly distributed system using 5 languages on landscape spread over frontend and backend with three different db engines and two clouds ? Kubernetes is easier to start but 10 times as hard to do at scale you didnt even have back then in OnPremise era. Yes you are right there are a lot of ppl who can write and deploy „Hello World” apps, but thats about it.

winternett 3y ago

Since the in house server days, technology solutions have only become more and more time consuming to implement, ID & PII obsessed, and bulky (data footprint-wise). I am thankful I remember much more simple times, when apps were tiny, run on regular computers in my office, and you could symply restore from a local backup to bring a service back online. Now we have companies paying thousands of dollars a month to host a simple site or web app, because now it's all in the cloud, no matter the apps function, while hosting customers assume about 80-90 percent of responsibility for securing and maintaining their app anyway. Containerization is leveraged to add a "backdoor" for individual admins, but that only serves to protect infrastructure, it actually makes a customer's individual app open to vulnerability that same as running your own instance if you miss vital updates or if there is a breakdown in the supply chain. We need to stop listening to the companies that market tools and be honest with ourselves... Security needs a better model than just adding new tools and entry points to the app chain. There's gotta be a point where you evaluate things and tell yourself that there has to be a better and more affordable way than tunneling through 5 VPNs just to push updates for 8 disparate JS and PHP libraries and 5 containers. App and library updates are also far too frequent in 2023 as well... Frameworks need to ship instances with less features, and modules should be reduced to only those used and deemed most essential, reducing footprint is a key aspect lost on technological advancement, it is also a firm indicator of increased efficiency for app and library devs if you ask me. Now with zero trust, devs literally spend a lot of their daily workload logging back in and re-establishing VPN connections due to timeouts. There's got to be a point where we develop a far more simple IT solution to all of this mess. Security is still getting compromised regularly no matter what is done. The solution also won't likely involve Ai at this point in my opinion, as security and complexity are assuredly not resolved by leveraging current-state Ai tools. It's also important to note this is why PHP is still going strong, it doesn't require compilation, and is relatively easier than most other langs on learning curve to implement. I literally built my IT career on making things simpler, and in explaining complex IT issues in human language to non-technical people. There is a lot of room in my field for growth due to the rest of the industry's constant focus on jargon and over-complexity. Simpler solutions, conveyed and implemented in human language, will be king in 2024.

xnx 3y ago

ChoHag 3y ago

[dead]

KaiserPro 3y ago

I think the biggest issue with devops is that it originally meant "socialising" your sysadmins, by getting them to sit with your devs, so that shit didn't get lost because nobody thought to talk to the right team. But then it morphed into "oh lets innovate with infrastructure" but the innovation turned into "lol lets just restart from scratch and ignore history" Anybody who used early k8s can attest to how un production ready it was for the longest time. It _felt_ like progress because it required a lot of (to dev eyes) hidden magic to make it work. Now, we are back with sysadmins, but they are called SREs. They provide a platform, which devs talk to to deploy stuff. Basically we're back where we were in 2015, just with more yaml. The good things that have come out of devops has been the dashboarding and metrics tools. Its just a shame that everything else touched by it appears to be an infinite source of busy work (looking at you kubernetes) (before anyone asks, I'm an SRE at a FAANG.)

notanormalnerd 3y ago
So you mean, DevOps has made Sysadmins more service oriented and now instead of having their pets and being in overprotective silos, they actually provide best practices and proper platforms for the poeple developing the software?
- KaiserPro 3y ago
  thats how it _should_ be. The place it worked the best was where a team of sysadmins were split up and embedded with each dev team. each devop would then rotate onto another team every 6 months or so. This meant that _we_ had to document our shit for the next devop, but also helped eliminate key person dependencies. We would then have a weekly offsite where we'd bitch and moan about our teams and conspire to make things better.
- sisve 3y ago
  Im not sure how former sysadmins feel about it, but from a dev standpoint, DevOps worked great for us. Making a clean cut on a plattform. Was my app down, I got message/called. If the platform was down it was the Infrastructure team / PaaS provider that needed to fix it. In the old days we had so few deploys and a hotfix was something that was extraordinary. I actually had a board of something in another country approving that we where allowed to deploy. They knew nothing about the application..

mschuster91 3y ago

Yuck, Portainer... when I read that I get nightmares. The problem is: when you give developers access to Kubernetes, to ECS, to Portainer, to whatever they. will. not. care. about literally anything. You'll find a hotpot of cobbled together Dockerfiles with zero provenance, with base images pinned to stuff years old, and completely inefficient layer ordering. Pipelines won't use caching or parallelism (okay, sometimes because they use Maven which is a clusterfuck on its own). Software developers are developers, 90%+ will never have heard of how Linux works below the hood, how Docker actually works, they'll just copy and paste together stuff from Stackoverflow without even attempting to understand how it works. And I don't even blame them. I can't, I don't want and I won't. Companies, it's time to stop loading stuff onto your developers that they don't have to (have) any clue. JFC.

ElectricalUnion 3y ago
The developers are literally paid to do the least amount of work that could possibly do towards delivering some abstract MVP "this sprint". Actually making something that is simple and maintainable is the 7th or 8th priority of business, if it is a priority at all. Artificial "obstacles" to delivering the artifacts in time in fact, wanted, a sort of corporation Deus ex machina, externalities to explain why they did not meet delivery deadlines last time. Incentives are not aligned. Things will remain broken until incentives are aligned.
shrimp_emoji 3y ago
And some devs do learn how Linux works but don't want to waste their time memorizing how shitty $DEVOPS_PRODUCT that'll be replaced in two years works. :D
- mschuster91 3y ago
  Indeed. Which is why I waited for three years to take on Docker, and two more for Kubernetes after I got burned with DC/OS, and why I entirely left the NodeJS/frontend ecosystem. I'm too old to act as the ripener for bananaware any more, and I'm pretty sick of companies exploiting other companies or, even worse, individual developers to act as their free QA department.

lifeisstillgood 3y ago

Nobody knows anything The difficult paradoxes have not been resolved No-one can say this loudly because there is no safe space

lifeisstillgood 3y ago
Nobody Knows Anything For a given level of complexity in an organisation, talk to any 10 experts on the subject in the organisation (ie how does the foobar process work) and you will find blind men describing an elephant. All technically correct but all missing important parts The paradoxes At some point an organisation acquired or grew an antogonistic part. Both cannot really exist but both have not been removed. Think either Google search vs Google ad sales (if you return perfect search terms each time, you get less page views hence less ad impressions. Make search worse get more ads served). Or simply other non core businesses. Pretty much every non-tech mega corp is really dozens and dozens of smaller businesses with a shared treasury so they can ride out each others ups and downs. But one or two businesses are really carrying oat of the load. Anyway, you can find paradoxes or conflicts of interest in all businesses and badly aligned incentives hurt everyone. Someone should fix that. Not me I don't have the power (indeed usually only the orignal centralmfounder has power to) Safe spaces Yeah these don't exist. If you ain't upbeat publically you are seen as trying to pull everyone else under. So how are problems acknowledged let alone fixed? Dunno. What would really help would be free speech, open conversations and discussions based on data that can inform policy and provide impetus to chnage. we (kinda) have this in western journalism / media (including social media). But absolutely no large corporation anywhere has a open discussion area where different factions can raise their concerns. there are of course political infighting amoungst the powerful elites of the corporation but that's not the same thing - they might even be accurate about the problems but no one else knows. Democracy matters Sorry this was supposed to be about devops - and it kinda is. Most large corps have some devops thing going on where "this time we will fix it all for everyone" - but has there been open conversations with data supporting it from all parts of the org? oh come on.

k8svet 3y ago

"DevOps" as in Developers being closer to Ops, or "DevOps" as in "I'm a shiny Golang outfit that is going to solve Real Problems TM with YAML and a bunch of templated-generated Golang?" I realize this is a bit of an odd tangent, but I would love to hear cadey talk more about what "devops" means at a shop like Tailscale. Actually, as I type this, even more curious. I know some of those folks are cut-me-and-I-bleed-Go types, but I also know they're deep into NixOS for server management. It would be very interesting to hear more about how those intersect for their infra/server/dev culture.

dilyevsky 3y ago

Most commenters here seem to ignore the fact that saas deployments have become increasingly complex over last decade in part bc low hanging fruit has been picked. Also the massive scale of even relatively no-name companies out there. I'd add that most of the 1-4 "problems" (boo-hoo, managing IAM is boring for devs) OP mentioned as well as advent of so called "platform engineering" stem from one thing only - poor engineering leadership.

moron4hire 3y ago

What drives me nuts is the number of places that expect you to have deep experience with whatever CI/CD solution they're using before they'll even consider you for a position. How have we gotten to the point where deploying an application is so fucking arcane of a process that you need to have 5 years of experience with whatever acronym soup some CTO bodged together or you are considered worthless?

pdimitar 3y ago

Replacing "DevOps" with "Platform Engineering" is not waving a magic wand and removing problems. The original article's main point is: "use DevOps marketplaces", more or less, but that comes with its own set of headaches (mainly auditing and making sure you trust them, which might easily take more time than rolling your own stuff). I fully stand behind the "just have programmers that write internal tooling" recommendation though. Over the course of my long career this is what has worked best almost every time, and these are the programmers who get the least amount of respect even though they have to dance and juggle on a burning rope more often than not.

fmajid 3y ago

Someone at a Linux company once said: if you combine "Terraform" and "Ansible", you get "Terrible".

jiggawatts 3y ago

In my opinion, more than half the problem is the tooling, as indicated by the cartoon in the linked article. DevOps pipeline languages are generally either weakly typed or stringly-typed. They generally expect users to do the control character escaping, in several different obscure formats. Did I say escaping? I meant three layers of nested escaping! Variable/macro substitution as a rule uses syntax that overlaps with either PowerShell, Bash, or both because otherwise it would be too easy. Another basic thing is the absence of a debugger — I have to run a complete pipeline with a change to dump out environment variables! Instructions contain the display names of paths, scripts use the variable names, but the output (and errors!) use the values with no clear mapping backwards. What the eff is C:\s\1\x? I dunno, but it caused an error! The worst sin is not having a local development experience where the entire pipeline can be run without having to commit code and push stuff almost all the way to production. Even if the pipeline agent is made available, it’s useless without the underlying VM image which is huge and not easily obtained. Not to mention KMS/KeyVault and other similar services on which pipelines can depend. Developers complain about having to do DevOps, but in my opinion the problem isn’t who is doing it, but how. The tooling is just so bad, so very very bad, that anyone would hate it if they were the ones forced to do it. I’m an ex-dev-now-SRE and I hate it.

cdnsteve 3y ago

Isn't this what PaaS is suppose to solve?

aloknnikhil 3y ago

This is the original author / article - https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88162c86d7 https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88...

hgopolis 3y ago

Sysadmin, devops, platform eng all suffer from made up problems. What’s IAM? Access control. What’s a firewall? Access control. What’s a route table? Access control. What are resource limits and reservations? Access control. Yet years of legacy solutions has us discussing all these things as if they are fundamentally different. User management comes wrapped in vacuous jargon. Network addressing schemes are completely design by the pocket protector committee (fuck off they can be wrapped in abstraction that is less obtuse jargon and more memorable/mnemonic). There are no users, groups, or network routes in the machine. Devops is obtuse access control because a lot of companies can make money from insuring its obtuse access control.

rswail 3y ago
Calling a route table "access control" is not useful. It's not access control, it's literally in the name, it is about navigation of the network via routes. As for your comment about the "pocket protector committee" that's just ad hominem attacks on the people that keep the internet running. Saying that NDP is more complicated than ARP/RARP etc because it has a different name is, again, attacking actual needs because it's more complicated than the protocol it replaces. A network ACL is access control (it's in the name), L2 VLANs may be used for network separation, but they're also used for switches to be able to not have to broadcast packets across all their ports. If you want to downplay the different requirements that the different forms of resource management require as "access control", then all you're doing is removing all meaning from the term "access control" in the first place.
- hgopolis 3y ago
  I design motherboards for network devices; I keep the internet going. Routes are inaccessible without access to them. The underlying machine states to check for access to a route is algorithmically similar to the other checking for user access. It’s unfortunate so many are allowed to work in IT with cliff notes level awareness of how technology works. The layers of indirection you work with are the knobs and buttons people like me choose. You merely obfuscate with vacuous jargon to serve your career goals.

skyrocketer77 3y ago

The original article is here, this guy just copied it, lol https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88162c86d7 https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88...

donutshop 3y ago
Underrated comment

high_5 3y ago

Devops got stunted or intermingled with the great migration to the Cloud. It could be practiced on any level of infrastructure, but the discourse currents of the Cloud were too strong.

avryhof 3y ago

Where I work, it just means I have to maintain my own infrastructure in addition to being a developer.

nunez 3y ago

2023 DevOps is awesome. Today's "sysadmins" are kind-of, sort-of expected to know a programming language, at least superficially. This was definitely not the case in 2015. The whole idea of Platform Engineering formally existing (I.e. platform is a product) is fantastic. There has always been lip service from IT about treating the rest of the business as customers, but the definition of "customer" was usually bastardized to fit whatever agenda IT execs wanted to push. Not the case (as much) anymore. Also, the idea that today's "sysadmins" are part of application delivery into production was extremely nascent in 2015. Most sysadmin jobs didn't touch apps with a 10-ft pole back then. This, IMO, is the greatest benefit the movement brought to bear. Shoot, even the fact that saying DevOps feels dated because it's obvious now demonstrates how far we've come. Saved the best for last. Kubernetes is hella complicated compared to 2015 "DevOps" tech stack (which i miss because they are simple), but provides so so so SO many benefits for free, especially if you're not in public cloud.

dijit 3y ago
> Today's "sysadmins" are kind-of, sort-of expected to know a programming language, at least superficially. This was definitely not the case in 2015. There was a "weird period" in 2010-2014 where mid level sysadmins didn't know much scripting or automation. I'm not sure how that happened. Senior sysadmins always scripted, Ruby was the de-facto choice for those folks in 2010-2015; before that it was mostly perl, and after that it seems to be mostly python. Mid-level sysadmins definitely knew bash and perl in 2000-2010. I think it was the rise of calling level 3 helpdesk "sysadmin" that sullied the brand. But the same will happen with other disciplines. I'm sure I'll hear in a couple years about SRE's or DevOps who can't code. Heck, I bet there are already devops who can't code, who only know how to template yaml.

donutshop 3y ago

Looks like OP copied and pasted the entire article from somewhere else. https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88162c86d7 https://medium.com/@mbianchidev/2023-devops-is-terrible-ec88...