12 ms·

DevOps Is Bullshit (2022)

• dijit 3y ago
  In looking for "past" articles from this I stumbled upon this from 8 years ago: https://news.ycombinator.com/item?id=9170615 https://news.ycombinator.com/item?id=9170615 It's interesting how differently we discussed DevOps back then. EDIT: also ThePrimagen reacted to this article yesterday: https://www.youtube.com/watch?v=qVEEpUvl0Kw https://www.youtube.com/watch?v=qVEEpUvl0Kw
• ThalesX 3y ago
  I'm surprised recently, just how many job offers and what compensation my devops friend is getting (he's been a devops for almost a year... before that he was tech support, very basic technical knowledge becomes he comes from outside tech). My senior full stack developer LinkedIN requests are pretty low volume for some months now, while he gets flooded with offers with compensation more than most senior devs I know. His skill set is also (and I love the guy dearly, I'm trying to be objective and he also feels this way) maybe 1 / 5 of what a senior dev has. Basically he's "I’d wager most of what they are doing is using Terraform and YAML to do menial tasks for the engineering team."
  • heinrichhartman 3y ago
    If this is true, why are you not moving into "DevOps" roles as well?
    • adammarples 3y ago
      Because who wants to be a yaml engineer
    • JohnMakin 3y ago
      It is true. People hire astoundingly incompetent devops all the time. I’ve seen it throughout my career. An issue is most hiring managers and CTO’s also don’t have a devops background, so it’s trivially easy for people to get into situations where they can BS some buzzwords and googled regurgitated definitions like CI/CD and it’s like “welp you know what you’re talking about.” It’s highly compensated because it’s very needed. A low low % of operations engineers are as good as they need to be. It’s not even always the organization’s or the engineers’ fault, either, as the author of this blog alludes to.
    • kachnuv_ocasek 3y ago
      Because no amount of money can compensate for the soul-sucking pain of being a YAML engineer.
    • tirpen 3y ago
      Do you base all your decisions about life on a pure money maximizing algorithm? Picking a horrible position just because of a better salary is a great way to live an unhappy life.
      • mm007emko 3y ago
        I have a shit job (Python DevOps in tel. co. industry). No problem with that. Previous job was in applied research. Very interesting, unfortunately the work conditions and compensation were much worse than my present-day shitshow. Life is not only about work, I have to work to be able to pay my mortgage and to enjoy other activities. Having happy or unhappy life, for people in the IT industry, is mostly about their mindset.
        • Detrytus 3y ago
          If that lifestyle suits you, sure, go for it. However, by staying in a shit job you are throwing half of your waking hours away. I know this is not an option for everyone, but if one can make enough money from the thing that one enjoy that's like getting half of your life back.
    • awesomeMilou 3y ago
      Because the work conditions outside FAANG are garbage, no one that has any regard for their own well-being would get into it. So far everyone I have met that works in infra was in horrible shape, often from stress-related eating disorders and lack of physical exercise.
      • ThalesX 3y ago
        My friend works outside of FAANG, in Europe, and the DevOps department is super chill.
    • ThalesX 3y ago
      This question is strange honestly... Because I have a career in software development and I really enjoy doing it? If I wouldn't be in this situation, maybe I would consider doing the switch based on what I've seen.
  • ineedausername 3y ago
    True only to some extent, the knowledge you need to be a competent devops is huge and i think most techies prefer dev route though i don't know the stats.
  • awesomeMilou 3y ago
    The issue lies exactly with the menial tasks. Ever used Ansible? It's hell. Not because it's hard, but because there's basicially no debugging and so much hidden state. You're lucky if you can use Terraform, but that's not the default. Depending on the stack, DevOps is mainly just trial and error, "getting shit to work", which depending on the Software used can take a while. And then being responsible for the part of infra you got working, which may necessitate unpaid overtime in case of an outage. EDIT: One thing I forgot is, that in order to be really good at DevOps, you need to know your stack _really_ well, which imo necessitates that you spent at least some time on the development side of things. This goes against every hiring managers instinct though, as people should only specialize in one thing and do it well (at least where I live).
    • nailer 3y ago
      > You're lucky if you can use Terraform, but that's not the default. Oh god I thought that was unlucky. I have huge respect for Terraform for popularising what they call “infra as code” (we used to just do this with the actual AWS API) but now there’s multiple modern IaCs that use actual code with real tooling.
      • shyn3 3y ago
        What's your favourite?
        • nailer 3y ago
          I still like the flexibility of the raw AWS API, but now days CDK and Bicep for anything more complex. I really wanted to like Pulumi but it had a lot of odd bugs and leaky abstractions a couple of years ago when I got into it, it’s probably improved since then.
      • awesomeMilou 3y ago
        IaC was popular before Terraform, it's just the first tool that really made it easy to use. Unfortunately you can't make use of it outside of $BigCloud, because it needs common and well established API's to run. If your employer self-hosts, you're either stuck with Ansible or you host OpenStack yourself (which is just a surrogate $BigCloud, since you'll likely employ the services of RedHat to manage it).
  • rajamaka 3y ago
    As a former DevOps who has moved into SWE, I completely agree. One consolation I might put forward is that there's a larger breadth of knowledge with much less depth when the two are compared.
  • jstx1 3y ago
    There's lots of moving parts, some systems are very complex and it's a space where experience is worth a lot - it's hard to get started with ops 101 just by yourself like you can do with frontend web development for example.
  • dumpsterdiver 3y ago
    As someone who transitioned from a DevOps role to a Security Engineer role, the thought of going back gives me anxiety. Having my "log light" turn red at 2am? No thanks. It's not like it woke me up or anything, I'm always awake at 2am, I just don't want to deal with servers at 2am. I'd rather get a good night of sleep and go in the next day and destroy all of the hard work that the DevOps team spent so much time working on. That's a much better fit for me. I wouldn't take a DevOps job even if it paid $50k more than what I'm making now. I just wouldn't enjoy the work. That isn't to say that I'm unwilling to help the DevOps team (I help them all the time by hacking their shit), but I wouldn't actually want to be them again.
• sirwhinesalot 3y ago
  I have no idea how DevOps morphed from having the Dev team and the Ops team working together instead of throwing shit over the wall to a job description that neither side wants to have, yet here we are.
  • jetbooster 3y ago
    I feel like DevOps should be a skill to mention you have experience in (Ops with experience of Dev or vice versa) but somehow morphed into a job description all on its own, defeating the objective of the term completely
    • jen20 3y ago
      This happened because people accepted the linguistic uncertainty, pushed by vendors who wanted to sell stuff. The well may be poisoned, but the original meaning is still perfectly well alive.
  • zo1 3y ago
    I can tell you how I saw it happen first-hand from my "biased" viewpoint. It was 100% career-bullshitters within the org wanting to make a name for themselves by "introducing" something that the clients wanted and was a "big gap" for the consultancy. They latched onto this buzzword, and at the same time it was combined with clients latching on to it as well from all the hype it was getting in conferences, talks, articles, and other places where bullshitter evangelists push their drug. Never mind that in every sales meeting with clients we "promoted" this buzzword - so no wonder they "asked" for it. Had one of my projects burning for a year as a result, where we were told by leadership to rely on a "platform engineering" team that had a ready-made platform. Meanwhile it was just a bunch of arcane priests that knew some awful-looking JS/JSON inside a YAML templating language, running on some weird platform tool that manages K8s. Found out about the mess when I inquired after a month where this ready-made "platform" was, and they said they were still "bootstrapping a K8s cluster in AWS", then waited another 2 months for them to "bootstrap" the QA and production environments. Meanwhile the platform engineering team was led by a career-bullshitter and they had just a bunch of guys that basically did some bootcamp courses, certs and minor experience in "Their Chosen DevOps Tech". Leading to such wonderful talks summarized as: "Oh you want to explore the Hashicorp stack and weigh pros and cons? Fuck you, it's a poor-man K8s, we know better, shut up and stay in your lane." Never again. From that point on, this function 100% stayed within the dev team's control. Architecture, platform-ing, SRE, devops, whatever you want to call it.
• frank_bb 3y ago
  [dead]
• nailer 3y ago
  Well, yeah. 25 years ago, there was a thing called a sysadmin. A sysadmin was an infrastructure engineer that knew how to code to automate everything. There was SAGE and various UNIX and linux groups. A lot of Windows people disliked the concept of Sysadmins because they didn’t know how to program and tried to paint the concept as elitist. A decade later someone announces the concept of DevOps, which is an infrastructure engineer that knows how to code to automate everything. Whatever. it’s not worth getting worried about. At least automation is being considered important again, we have infrastructure as code now, the languages are better than Perl, sendmail is dead, and as long as you stop them installing kubernetes unnecessarily (which they will try) they’ll be fine.
  • jen20 3y ago
    > A decade later someone announces the concept of DevOps, which is an infrastructure engineer that knows how to code. … except that’s not what the concept was.
    • nailer 3y ago
      …to automate everything. I wrote it out fully the first time, and didn’t think I’d have to write it out the second, but hey, there’s always one person. If you don’t think devops is about automation… honestly I actually don’t care have a great day.
  • nuker 3y ago
    > A sysadmin was an infrastructure engineer that knew how to code to automate everything. … on a metal server. In Bash.
    • sgt 3y ago
      Bash? What a luxury. I mean we had bash, but all our scripts were made to work with plain old /bin/sh
      • nailer 3y ago
        I still save non-sh compatible bash scripts as .bash and my colleagues have no idea why, because they don't know Bourne shell is a thing.
• arghwhat 3y ago
  The problem with DevOps is not really how it is practiced, but how it is named. DevOps is a bucket that a bunch of things are thrown into, and everybody thinks the thing they threw in is DevOps with the others being mislabelled. Is having a DevOps team wrong? No, having automation experts, infrastructure experts, integration experts, etc. all makes perfect sense. Bundling all those disciplines under a single term is what is wrong. Is aiming to do "less DevOps" using "serverless" wrong? No, aiming to reduce infrastructure maintenance and design overhead is perfectly sane. Interpreting that as entirely avoiding infrastructure, or as affecting any of the other disciplines is what is wrong. What we need is better work descriptions and team organizations. But while the buzzword DevOps is bad, it's not so bad to have a fancy term that executive management understands and can put some money behind - money that allows you to spend time or buy solutions to solve your actual process problems are.
  • johnnymorgan 3y ago
    Interesting point and one that I feel applies equally to Product Manager, another catch all title that has a ton of stuff thrown in. IMO there are 3 distinct roles that product gets thrown into, 4 really but the technical product manager is completely different. Research/data, operations and UI/UX and then the 4th being the technical product manager. The person that maps and sets up the metrics, KPIs, product intelligence and knowing the business model is very different from the person that knows the UX flow and can inform the UI. The problem IMO lies in the middle management layer, maybe ai will be able crack that but for us ;) It seems to be a problem all over many disciplines tbh
    • deleted 3y ago
      [deleted]
  • wouldbecouldbe 3y ago
    Having an automation expert would either hint to very unsuccesfull automation, or a very very large org.
    • jhot 3y ago
• nicoburns 3y ago
  I think "ops" is ultimately like any other role: a specialism for which it makes sense to hire specialists (at least in larger organisations - people in smaller companies may have to wear many hats). And like any other specialist roles: close collaboration across specialisms is key. Application developers ought to have a decent working knowledge of ops such that they can work productively with the ops specialists and vice-versa. Just as application developers ought to have a decent working knowledge of product and UX design in order to work productively with specialists in those areas.
• deleted 3y ago
  [deleted]
• edwinjm 3y ago
  Also see this "reaction video" by ThePrimeTimeagen: https://www.youtube.com/watch?v=qVEEpUvl0Kw https://www.youtube.com/watch?v=qVEEpUvl0Kw
• elated_turing 3y ago
  Posted 8 months ago, discussion: https://news.ycombinator.com/item?id=33274988 https://news.ycombinator.com/item?id=33274988
• ozim 3y ago
  DevOps team is when dev team has dedicated ops or sysadmin that has all credentials and works with the team on "best practices" so that devs don't reinvent BS. There is no DevOps role and never was. It was just management twisting it into marketing scam to get devs operate systems and to feel like it is their responsiblility. Problem is that no company wants to have Sysadmin tied to one dev team because sysadmin won't have much to do most of the time - but that was idea of devops that you should have devs, admins, qa, business analysts working together reviewing tasks from the start and giving input on what is their specialization. Well yeah no true Scotsman - but dev ops idea is not bullshit - way people do things is bullshit.
  • awesomeMilou 3y ago
    > you should have devs, admins, qa, business analysts working together The idea of DevOps is to my knowledge that the role of Developer and Operator slowly merge together in one role. Which would also be in line with SCRUM and agile development, i.e everyone should at some point be able to do everything in the stack, if it's necessary. What you're describing sounds more like the standard agile development process.
    • adamhp 3y ago
      And the growth was really driven by the great exodus to service oriented and microservices architectures from monolithic webservers deployed to Tomcat and the like. The author in the OP really nails it when he talks about the ever-growing complexity of modern tech stacks. It necessitates this blurring of the lines.
    • zokier 3y ago
      > The idea of DevOps is to my knowledge that the role of Developer and Operator slowly merge together in one role That is certainly not how devops was originally envisioned.
      • dijit 3y ago
        People downvoting this guy are misinformed. There are two major sources for DevOps: 1) the 10+ deploys a day talk from Flickr: https://www.youtube.com/watch?v=LdOe18KhtT4 https://www.youtube.com/watch?v=LdOe18KhtT4 which is where most of the actual methodology comes from. 2) The "DevOps Days" (first use of the portmanteau by Patrick Dubois) which was an agile conference aimed at Systems Administrators trying to get them to work more agile; the conference was called "DevOps" to make developers think it was also for them. Neither of these primary sources said that we should erode or replace ops people; or that it was ever intended to be a role or individual responsibility. This is a lie that corpo's have made to reduce headcount.
• JohnMakin 3y ago
  This is so well said and on the nose that it hurt to read. I am entering into my third major project in a row where it’s almost the exact same situation (and same consultant I’ve bumped into) - site starts to scale, never had an ops team - realizes oh shit we cant manage this ourselves but who wants to hire some hacker nerd for $150k a year that “gEnErAtEs nO BuSiNesS rEvEnUe” so they hire a consultant or consultancy firm - firm builds them a solution that will technically “scale” but no one in the organization bothered to learn how it works - it works for a while til it doesnt and then the “oh shit” moment becomes “oh shit we need a dedicated devops team” - devops team attempts to work with the shitty consultant solution (editing yaml) and does an okay job until something breaks horribly again or they become a massive bottleneck - finally they allow someone to come in and clean house and you get an infra platform like the author of this blog is describing Platform engineering is the right word. “devops” has literally no meaning. I’m only 6 years or so into my career, but I can count on a single hand the amount of devops engineers I’ve worked with that can actually “dev” - and by dev, I mean do stuff like dig into application code and suggest modifications for the infrastructure, or write their own full fledged libraries. Let’s get rid of devops and hire platform engineers. The problem is it’s a really weird and rare combination of skillsets, in a field no one really wants to do except complete weirdos (I am one). I’ve long said that DevOps isnt a career you choose, it just happens to you. Well said! thank you so much for this post.
  • dijit 3y ago
    20 years ago, when I was trying to enter the field, sysadmins were almost all former developers who had a pretty fundamental understanding of operating systems. Somewhere along the way "sysadmin" became a euphemism for helpdesk, and some folks thought "devops" (as a job title) was new, but it was just the same old sysadmins really, or new devs becoming new sysadmins with the fresher title, they felt themselves the vanguard of a new paradigm, I'm sure. Now we're in the same situation again, where "devops" (as a job title) can have no development experience. So we need a new term, apparently. Maybe gatekeeping is useful to stop this job title treadmill.
    • JohnMakin 3y ago
      I think part of the reason this trend has been going on, and I have observed a similar thing as you point out here, is that businesses have tried to get rid of this “role” for a long time now and by constantly reinventing and renaming it. They have failed miserably. When people ask what I do, I call myself a systems engineer or “backend engineer” if that makes more sense to them. it’s more apt for what is actually needed and asked of me.
• DonHopkins 3y ago
  When did sysops stop being programmers?
  • dijit 3y ago
    sysops: 2003 sysadmins: 2010 devops: 2017 platform engineers: 2024 Wonder what the next term will be for developers who understand operating systems and distributed systems holistically.
    • pcthrowaway 3y ago
      Webmaster: 1997
    • jraph 3y ago
      Deep-Stack Engineers? (Full Stack, a misnomer, is already taken)
      • DonHopkins 3y ago
        An inside-out stack is a hole, so Deep-Hole Engineers: just keep digging!
    • sam_lowry_ 3y ago
      Hm... Sysop is a term from the 80-ies. Sysadmin from 90-ies. It survived 00-ies and Devops became a thing in 10-ies.
    • DonHopkins 3y ago
      DevOps + Object Oriented Programming = DevOops
      • datavirtue 3y ago
        Winner. Two misunderstood ideas.
    • campers 3y ago
      Full-cycle developers? Afaik the term was coined at Netflix https://netflixtechblog.com/full-cycle-developers-at-netflix-a08c31f83249 https://netflixtechblog.com/full-cycle-developers-at-netflix...
• ranting-moth 3y ago
  DevOps does not mean that developers are also Ops, like the article seems to be suggesting. That would be a bit like expecting your house builder to also architect your house. That's not going to happen, but they need to be able to communicate and understand each other.
  • Detrytus 3y ago
    No, it means that for most companies Ops roles are redundant. That's why tools like Ansible or Terraform, and the whole Cloud Computing emerged: so that infrastructure can be abstracted away, and you, as a developer who wants to deploy their code to production are only required to create a couple of simple yaml files, trigger the deployment pipeline, and be done with it. Ops roles are outsourced to AWS or other provider. I know, that's just a theory, real life is nowhere near that ideal state.
    • JohnMakin 3y ago
      because it’s incredibly difficult to build, not to speak of maintaining it. I have seen things close to this, but can’t quite get to the finish line. I’m sure some people are doing it, I guess, but I have yet to see it. a lot of times a team will architect a good, automated solution that works for 3 years untouched and then fails catastrophically because it wasn’t maintained or assumptions start to fail. this dream of self service devs is becoming a myth. devs do so much incredibly thoughtless stuff to infrastructure all the time. you need an infrastructure person in the process to make it resilient and allow them flexibility to do the stupid shit they want to do - with guardrails.
• huijzer 3y ago
  > During a sprint, an engineer adds `convert` (ImageMagick) to the mix to support manipulating images and forgets to update the Dockerfile, and then production goes down. Couldn't this be caught in CI relatively easy? Either CI is not running in Docker or the logic wasn't tested
• jhoelzel 3y ago
  om my god this guy is full of stuff..... >> "The growing zeitgeist is that “platform engineering is the future.” And given that I co-founded a product in the space, I sure hope so!" I am so sick of these shoesalesman trying to get you to adopt their platform which runs on another platform which runs itself on wishful thinking and startdust. Kubernetes can be deployed with a single binary nowadays there is no magic anymore the terraform scripts do look similar because you are not doing the actual hard parts like cis compliance or even selinux. This whole discussion is so moot, that every time it comes up i feel a little bit sad inside. Call the things i do what you want, call us the "server-clowns" who cares. It doesn't matter if its "platform engineering" "principle engineer" "devops dude" its the whole shebang of "guy with experience builds stuff that code runs on". You dont get people who do "DevOps" fresh out of college or people that have been barely using a computer for some time. You get the dudes with experience, but every time someone with experience makes up one of these buzzwords like "platform engineer" "lord of the box" or whatever, tons of people just assume the role without having the experience. And then you hire the cheapest ones and wonder why you need 15 of them. None of this is hard if you have been playing with these concepts for ages and finally that's what companies pay you for. And having your entire statup on serverless? Give me a break. How can you tell that someone doesnt follow compliance or any other regulatory body in one sentence.... If you click on the username you will see that this person tried the same post 7 month ago and therefore i will say it again: Please let al bundy, the shoesalesman complain somewhere else.
  • urbandw311er 3y ago
    Well spotted. This comment should be upvoted - the article needs to be read through this lens.
    • dogleash 3y ago
      > the article needs to be read through this lens Would people read this article less critically without knowing the author's position? If so, then THAT'S the problem with devops (and every other software buzzword). If someone is reading online about ideas in their profession, why should they ever treat the ideas with anything less than maximum critical thinking and distrust? This is their livelihood, after all.
• throw0101b 3y ago
  Ben Rockwood (then at Joyent) had an interesting take that he presented as the keynote at LISA '11 ("The DevOps Transformation"), making comparisons to Shewhart, W. Edwards Deming, and Toyota (Production System): * https://www.youtube.com/watch?v=3KpPBnEtRj4 https://www.youtube.com/watch?v=3KpPBnEtRj4
• trabant00 3y ago
  All the words are bullshit: DevOps, GitOps, SecOps, CI, CD, zero-trust, Agile, yada, yada, yada. It's people with their passion, expertise and attitude that make a good IT pipeline. Thinking you can capture it in a word, write some blog posts or courses and then apply it to whatever team is pure folly. But then what would the consultants sell? And what would massive IT service companies attach their products to? Hype is the grease that lubricates the opening of the wallets. I've been a sysadmin, then a DevOps, now a Dev. I loved system administration the most but then it got a bad connotation for management, recruiters, etc. Like sysadmins did not do automation or wrote code. So the jobs went away and then I got jobs writing YAML and using whatever alpha version automation tool was hyped at the time. So in the end I said fuck this, I'm tired of trying to put garbage code that others wrote into production using what they (who had no Ops experience) think makes Ops work. I want to write my own garbage code and let others use whatever born-yesterday YAML templating engine or DSL is trending nowadays. As an Ops guy the only way to win is not to play.
• datadeft 3y ago
  X is bullshit, also X cannot be defined exactly so everybody imagine what X means to you and I know for a fact that thing you imagined is bullshit.
• jillesvangurp 3y ago
  It's not bullshit but a lot of companies are definitely overthinking, over engineering, and overspending here. I've seen the transition from before the .com bubble to where we are now. Things changed and improved quite a lot. But the ambition level also went up. Puppet, ansible, teraform, cloudformation, ansible, salt, packer, etc. you name it and I've been there and done that at some point. But without ever having taken devops on as a full time responsibility. I'm a developer that can do ops when I need to. If you know what you are doing you can actually minimize your cost and complexity a lot and get away with not having to actually do a lot. I am a startup CTO these days. That means I'm not bored. Devops is not a person, it's me in what little time I can budget for this among many other more interesting, lucrative or tedious things that I also need to take care off. I need my servers to run and not die on me. I need decent uptime and build automation. I need backups, alerts, and all the rest. I've seen it done well and I've seen it done poorly. I don't like it when it's done poorly and I know what that looks like. If I had the budget, I'd probably get a person to delegate to and invest more in this. But I don't. So, I go for minimalism and reduced complexity. So, good bullshit free devops is what I do. No mico-services in my company. We have a monolith. That's not likely to change any time soon. We can't afford the operational complexity and I frankly don't see the added value of having to have that complexity. I don't use kubernetes for the same reason. Using kubernetes to run 1 docker container is nonsense. We run nice simple docker containers. Most common cloud providers have easy ways to scale and run those. I used google cloud run for a while (cheap and easy), but our use of websockets and threads made using traditional vms and a loadbalancer a bit more flexible. That's what we've had for the last two years. I don't need a teraform script to create that for me and I just clicked that together in a few minutes. My infrastructure is fine. I do use build automation. We have a few lines of gcloud commands in there that tells the instance group to apply a new template with our latest docker container. It runs when we merge to our production branch. We use Github actions for this. Our master branch goes to a similar staging environment on every pull request merge. That's simple devops. We also have a few managed services, alerts, monitoring, logging (via elastic cloud), and few other bits and bobs. I'm not a caveman. But you get the idea, this is not something that we need to spend a lot of time on. Set it up right and you can do more fun things. Our uptime is fine. I can scale this endlessly. I can also setup a new environment without too much hassle. If that actually becomes a regular thing, I'll automate it further. But that just isn't a thing right now and not worth a minute of anyone's attention in this company. Automating things you only do once every few years is a waste of resources.
• qaq 3y ago
  This is part of futile attempt by management of creating spherical worker in a vacuum so there is as little specialisation as possible.
• tokamak 3y ago
  DevOps = reducing black boxes at the cost of Dev team load Platform engineering = building black boxes (abstractions) to reduce Dev team mental load at the expense of nobody understanding how to fix issues
• switch007 3y ago
  Platform engineering: implementing a half-baked PaaS with kubernetes in the worst way possible, over applying the MVP mentality and breaking things every few months because making something stable is boring Know how to say no? Awesome, that’s what you need to tell the devs whenever they moan about not having fancy things like metrics or logs (Sorry just ranting about my current gig)
  • moduspol 3y ago
    This has happened at a few gigs for me. Don't forget ballooning infra costs, with no one team being able to be held accountable because one team is managing the platform and the other is writing the code that is run on the platform.
  • x86x87 3y ago
    Kubernetes will provide work for years (decades) to come. People will try to unwind the mess we are creating now for a long long time.
• throwawaaarrgh 3y ago
  This is one of those rare articles about DevOps that isn't bullshit. The irony...! This part, while true, bothers me: "Simple, just hire some more frontend and backend engineers to develop a great internal PaaS with all the golden paths your operations team is architecting while you are trying to build your actual product that runs on top of it." I've been on that team, multiple times. It doesn't really work, because it's yet another silo, and it takes years to bear fruit. We shouldn't be developing platforms, we should be buying them. It's like building and running a factory so you can have an assembly line to make widgets. Most people who want to make widgets today actually contract that out to a company in China who already runs a factory where they adjust their tooling to make your widgets. That's what economies of scale and specialization do: you pay somebody else to do the hard, expensive thing that isn't a core part of your business. You get to market faster, you waste less money, and carry less risk. We're doing business wrong. You should not be building factories, you should be hiring a factory. Do I mean "hire consultants?" No. What we need doesn't seem to exist. We need companies who do white glove integration of a software product with a bespoke PaaS. Like Heroku, but with more attention to architecture, monitoring, alerting, security, ci/cd, etc. Someone who will give you the tools, train your people, do integration alongside them, and provide support. And they could do this on a cloud account that the customer owns, with IAM guardrails to prevent them fucking up the platform. There is no value added to your product by building a factory. You might reduce cost, eventually, if your business is huge enough. But there's a small number of such businesses out there, and yours likely isn't one of them. So let's start some platform companies and make those the default choice.
  • quadrifoliate 3y ago
    > Do I mean "hire consultants?" No. What we need doesn't seem to exist. We need companies who do white glove integration of a software product with a bespoke PaaS. Like Heroku, but with more attention to architecture, monitoring, alerting, security, ci/cd, etc. Someone who will give you the tools, train your people, do integration alongside them, and provide support. And they could do this on a cloud account that the customer owns, with IAM guardrails to prevent them fucking up the platform. I think one of the reasons this doesn't exist is that you are describing every single component of the software development process here. I have a personal theory that software developers have developed severe tunnel vision about code in the last 10-15 years. Before then, most developers had some knowledge about deployment practices, shell scripts, apache, that sort of thing. They roughly understood how something like a web server worked; writing a toy one was in many CS curricula. Whereas now software developers' number 1 skill set seems to be "I know how to solve complicated algorithmic problems". Which is fine and dandy, but it takes so much time to master that it drives out all other deployment basics that many of us acquired while tinkering with software. I am talking bare minimum here, by the way -- I have worked with developers a year or two out of college who did not know what a shell script was because they had spent their entire degree inside either an IDE or Leetcode. This, combined with the fact that complexity has increased exponentially in the operations tooling space means that you sort of need knowledge about operations knowledge in your software engineering team along with complicated algorithmic problems. You could do this by hiring some very expensive consultants like you describe. I think you could also do that by altering your hiring pipeline to de-emphasize algorithmic problem solving, and increase the importance of ops knowledge and testing.
• rmnclmnt 3y ago
  Good article, resonate a lot with my experience. When recruiters come by and say "I looking for a DevOps engineer" that's a big red flag: DevOps is not a job, it is a philosophy to be applied to all related jobs within the software supply chain! As idiotic as if people come by to present themselves as "Agile Engineers" or "Scrum Engineers"... However: I do not think the DevOps philosophy is bullshit, far from it. Not perfect, like Agile methologies, meant to be bended and adapted. But once a month I hear stories from legacy ITIL-based organisations, which are point-to-point mentioned pain-points from "The Phoenix Project"...
  • tstrimple 3y ago
    Not sure why you're being down voted here, but I agree. So many companies are building "DevOps" teams which is just another silo. Folks who understand DevOps should be embedded in the project team. They should be functional developers who also know how to automate workloads. They shouldn't be building and maintaining pipelines for 40 hours a week. They should be working on the product and one element of their role is automating the infrastructure, builds and deployment of their components. The companies I've worked for which grokked DevOps didn't have dedicated DevOps teams. They had developers who understood automation embedded in the project team and the DevOps component was just one small piece they were working on. In companies which struggled, they setup separate dedicated DevOps teams and hired explicit "DevOps engineers" which you'd submit tickets to in order to get your pipelines built and changed. One of these enabled the teams to accelerate development and releases significantly, and the other just felt like you had a traditional ops team that was more of a barrier to getting work done. I view building software delivery teams similar to (in my imagination) building a "special forces" team. You cannot afford for your team to be taking on a building of terrorists only to come up against a locked door so your team shrugs and says "guess we'll have to open a ticket with the door breaching team". You've got to have the "specialists" you need embedded in the team or your "mission" will grind to a halt waiting to borrow resources from teams with different priorities.
• xkbarkar 3y ago
  Isn't platform engineer just another name though ? devOps, cloudOps, devSecOps, site reliability engineer and now platform engineer. It feels like the need to define operations is a bit like if a dev role needed to be defined by IDE or language How about visualGolangDev or categorize devs after what kind of code is being written microserviceDev vs backendDev vs middleWareDev In all honesty, can we just go back to ops ? It was a fine name. Said it all and running k8s , a unix server or a ci/cd pipeline is essentially just operations. What is wrong with ops ?
• slyall 3y ago
  Every single "devops" post on HN ends up the same. About half of the comments are people saying their there exact definition of "devops" is the one true one ( Handed down from Heaven by Patrick) and if people followed that then it would be unicorns and moonbeams Another third of comments are Developers who dislike Ops and especially that Ops can't code [well] and would seek to eliminate it. They hate that Ops people keep re-appearing with a different name each time. They will lament that Sysadmins always used to be able to code "back in the day" Some of the remainder will be people (like the author of this article) noting that Operations is a distinct skills and that most devs don't want to do it. SO rather than trying to eliminate it perhaps we could actually do it properly.
  • mehagar 3y ago
    Rather than developers not wanting to do ops, I took the article to mean that - ops is sufficiently complex - The incentive to deliver features means that ops corners tend to be cut So the suggestion is to build a self service platform that encapsulates the complexity, while making it easier for developers to use.
  • newtwilly 3y ago
    Doesn't help that the post didn't define their definition of devops, and then they went on to use it in a variety of contexts, seemingly contradictory. As someone who is a software engineer, but doesn't necessarily keep up on the latest hype, this post made me head spin when I tried to read it casually.
• liendolucas 3y ago
  I think one of biggest problems the industry suffers from is HYPE. And we have HYPE all over the place: cloud services, frameworks, programming languages, tools, application architectures, etc. And to my eyes DevOps is another hyped thing. Not only that, many jobs require you to be "certified" in a cloud service provider (a la Cisco like in the 90's) or on some particular tools (Kubernetes?). Then we re-discover that the OLD way was really good, that actually there was no need for something that over-engineered, and we go back and start doing things the simple OLD way. After some time, the cycle re-starts. What is DevOps anyways? I don't know, but today it seems that is a word associated to fancy cloud services and tools to deploy applications, way way different from the savvy system administrator from the 90's & 2000's that knew lots core concepts as: DNS, LDAP, SNMP, HTTP (Apache), SMTP, NFS, iptables, etc. If I had to hire someone for today tasks I'd probably stick to the 90's knowledgeable system administrator.
  • bsaul 3y ago
    I agree completely. I remember very distinctly when tech transitionned from following "trends" to becoming "hype" : when iphone and apple marketing made developping ios app a hipster thing. All of the sudden you had articles in fashion newspaper talking about "how a mobile app developer desk looks like" ( saw it from my own eyes, it was next to the "architect desk" and the "fashion designer desk"). It certainly helped with the social integration of software developers (which is definitely a good thing), but it also created this bizarre connotation that tech had to be cool. It doesn't. Technology first and foremost has to work. Especially software. I should add that it's now becoming even worse : tech now has to be politically correct and inclusive ( aka: boycotting a database technology because its author don't support some US legislation regarding transgender rights would be a totally ok decision)
    • relaxing 3y ago
      Hype existed before iPhone. In the 90s and 2000s it was "here's what a hip web dev's desk looks like" and the transition to AJAX web apps. Before that, the switch to Java and XML and delivery via applets. Before that the switch from minicomputers to desktops and access to the internet in your home. Before that switch from slide rulers and doing the math in your head :) They all came with style section articles and a feature on the cover of Newsweek.
• pif 3y ago
  > When was the last time you saw a product manager high-five the ops person and say, ‘fast fucking autoscaler!’? It will happen when the optimization of the autoscaler will improve the revenue, by itself and in a quantifiable way. Otherwise, a fast autoscaler is just a small step in the right direction, as elegant code, clean financial accounts and free strong coffee are: all of them support the business, but it's the sellable functionalities that create the business in the first place.
• biorach 3y ago
  > In production, we are running in containers, but developing in the container is too slow, what? Why? > so the team leans towards asdf and a README full of stuff to copy, paste, and pray. It's really not that hard to run stuff in containers. Use a Makefile or similar so you don't need to type long Docker commands > During a sprint, an engineer adds convert (ImageMagick) to the mix to support manipulating images and forgets to update the Dockerfile, and then production goes down. And what... you can't run integration tests in a containerized environment in CI?
• solatic 3y ago
  I challenge the author to showcase a developer platform that handles secrets management and access management correctly and transparently. Sorry, can't be done. I've seen various attempts that certainly tried, and they've all had holes. Product teams need somebody who has the following skillsets: security, observability, deploy/release, containers/repeatable environments, networking/runtime/storage... if you don't have those skills on your team, you're going to have a hard time. Sometimes you find those skillsets in one engineer, sometimes among multiple people on the team. If they're in one engineer, maybe that guy's title is DevOps, maybe it's Platform, maybe it's SRE, maybe it's something else. Stop sweating the title and start hiring for the skillsets you're missing. If you can't find anyone, find people on the team who are interested in learning, and give them time to train on it. Who cares what other companies are doing, worry about the missing holes on your own team first.
  • Detrytus 3y ago
    > I challenge the author to showcase a developer platform that handles secrets management and access management correctly and transparently. In my my opinion Microsoft came pretty close with Azure DevOps.
• vbezhenar 3y ago
  The problem is that we miss simple and scalable platform for small-to-medium companies. Clouds might be that platform but many people don't want to lock in cloud API or just can't use clouds. Kubernetes is awesome but it's too low-level and needs to be accompanied by unholy mess of additional software and at that point it becomes too complex to be called "simple". My opinion is that the platform must be: Not built upon k8s. K8s is too complex by itself, too many moving parts, too many failure points. And everything built upon it makes it more complex. If you have team of devops, that's fine, they got work to be paid for. If you develop alone, that's a problem. Leverage containers and docker. Docker is a thing that everyone knows, anyway. It should be a ISO which runs in memory. Talos is a perfect example of this approach. It should solve distributed storage automatically. You run this ISO on few computers, their local disks become a distributed storage. It should provide overlay network for containers automatically. It should have lots of low-level features built-in and without choice. I hate to choose between calico+istio vs cilium. It should not happen. It should provide all the necessary metrics, centralized log storage, easily installable CI/CD pipeline, it should be resilient by default against crazy containers eating all memory or CPU or network or I/O. I guess, it could be built upon kubernetes, but it must be a very well chosen set of components without a way to replace those components and it must be extremely resilient and well tested set of components without any nonsense. With some reasonable UI dashboard, so nobody would need to install those grafana monsters, for example. With built-in OIDC, so no need to spend months installing keycloak and hooking everything to it. I saw some "platforms" built upon kubernetes, but those were duck-taped monstrosities which work as long as you pay their authors for 24/7 support and overvision.
  • jimmywatersabc 3y ago
    for small & medium sized businesses: a blue green deployment via CICD. For example, via a few docker containers & an nginx configuration pointing to those containers as upstream servers. https://www.redhat.com/en/topics/devops/what-is-blue-green-deployment https://www.redhat.com/en/topics/devops/what-is-blue-green-d...
• deleted 3y ago
  [deleted]
• gbrindisi 3y ago
  Same stuff can be said for all the devsecops bullshit vendors are trying to push down the throat of every organization.
• deleted 3y ago
  [deleted]
• harimau777 3y ago
  It seems to me that DevOps is just Taylorism. Trying to squeeze more work out of engineers by shifting responsibilities previously handled by systems admins.
  • firesteelrain 3y ago
    In my experience, back to sysadmin's don't code concept, we would keep it that way if self-service was a thing. Instead, we have standing IT Army's that gatekeep through extensive IT Helpdesk Tickets.
• oneplane 3y ago
  The entire article can be summed up the same way you can sum up bad security products: DevOps isn't something you buy or make a department out of, it's something you do, or better yet, it is a method. DevOps is fine, bad DevOps is pointless. Just like microservices are fine, but bad microservices are pointless. Same way bad metrics towards business owners causes you to seem to add no value (which goes both ways). That said, you can still find plenty of job postings for 'DevOps Engineer', which is part of the problem. But I suppose culture changes are hard...
• nunez 3y ago
  I disagree. DevOps was, and still is, extremely successful as a paradigm. Ignoring all of the other changes it brought into the industry (the fact that CI/CD is table stakes now, for example), the single biggest thing that the DevOps culture instituted is the _idea_ that ops will need to code sometimes and engineers will need to _ops_ sometimes. Sure, many engineering orgs translated this into creating a "DevOps" team that writes YAML and manages The Jenkins™. (At these places, this is, sometimes, the ONLY way that breaking silos can be done due to complex organizational rules. Also, many of these teams are composed of ex-sysadmins AND ex-SWEs, i.e. the literal embodiment of DevOps.) But it also brought about operators managing high-complex software like Kubernetes in even the biggest of BigCos. This simply would not have happened without the DevOps mindset taking hold. For example, from the article: > For every operations person without software development skills, there are FORTY engineers without cloud operations skills. If you are going to build an internal platform, you’ll need experts with overlapping experience in both fields working together. Yes. That's literally what DevOps _is_.
  • dijit 3y ago
    You've decided your definition, as many have. DevOps as written is not what you're saying; this has been mentioned at least 20 times in this thread. Sysadmins of yore were developers, they used perl and C and eventually ruby. Then there was a glut of sysadmins who couldn't code, then people acquired collective amnesia that sysadmins ever coded and invented a new term. The issue was that the term that they decided to use for this new breed of developer operations folk was the same word that was used for a methodology of breaking down siloes (having developers and operations as distinct entities but working together)[0] and the same word that was being used to describe sysadmins adopting Agile[1]. The annoying thing about the passage of time is that you look back and see only what has changed and may conclude that those changes are a result of paradigm shifts: you could be right, but sysadmins were using proto versions of common paradigms now: version control integrations used to be bad and fragmented, but it was ops pushing VCS; infrastructure as code via cfengine; cluster management with clustered ssh clients (cssh). I was there, I saw this. It's hard to differentiate between the slow march of progress of operations tooling (rebranded as devops tooling) and the fact that it might not actually be linked to devops as a methodology. [0]: https://www.youtube.com/watch?v=LdOe18KhtT4 https://www.youtube.com/watch?v=LdOe18KhtT4 [1]: https://www.knowledgehut.com/blog/devops/history-of-devops# https://www.knowledgehut.com/blog/devops/history-of-devops#
• 4BynHmBFUxBD 3y ago
  What happened with DevOps reminds me of what happened with agile development. The original concept was thrown into the development world, and it became a giant game of telephone. By the time it came out the other end it looked nothing like the original.
  • datavirtue 3y ago
    This happens to literally everything we touch. New mind blowing paradigm? It is almost always completely misconstrued, misapplied, and thousands of teams roll it out before understanding it and publish endless reinforcing blog posts about how to keep doing it wrong. If you don't want to do it wrong also, prepare to be persecuted and insulted.
• hestefisk 3y ago
  Can someone explain in plain English what platform engineering is? TFA spends a lot of time criticising devops but takes for granted that everyone knows what platform engineering is.
  • mehagar 3y ago
    The keyword is self-service. Instead of engineers filing tickets that say "please deploy my application", you have a dedicated team create an internal web app or something that lets engineers deploy their application themselves, but minimizes the risk (compliance, security, stability, etc.) associated with that deploy. This approach scales because it doesn't create a queue in front of ops.
• timcavel 3y ago
  [dead]
• efields 3y ago
  Point #1 is too real in my org. > Need a database? File a ticket with DevOps. > Need an IAM role? File a ticket with DevOps. > Before long, your massive team of engineers has fully saturated your understaffed “DevOps” team’s backlog. Our DevOps is under IT, not Engineering, so there's already an organizational gap there. I've talked with DevOps multiple times over _years_ and laid out a rough architecture of how Engineering would like to work. To this day after CI/CD passes its still manual deploys with a ticket. Render.com and other platforms do 90+%, if not 100%, of what I want as an engineering manager, but I'm hamstrung by policy and politics that were put in place 6 years ago when non-technology ppl decided we somehow had the resources to manage an OpenStack build instead of AWS. And that's before I realized everything on AWS is config config config and there's entire companies (like Render) that take away most of the work and just give you a dashboard to accomplish what you'd like to accomplish.
  • coffeebeqn 3y ago
    We have this exact setup with the same issues. Tickets for everything and their team is very understaffed for our team especially which is deploying new things fairly often
  • deleted 3y ago
    [deleted]
• Jochim 3y ago
  As soon as something becomes it's own team or department cooperation dies and a war ignites over attention and resources. My experience has been that the places that treated development roles as cross-functional have been the most efficient, with the least amount of politics and dysfunction. It removes a whole cadre of people who are barely able to describe the features they request, while minimising the space for power plays.
• efitz 3y ago
  I had a completely different experience with DevOps at a large MANGA than the author. Where I was, DevOps meant that the team that wrote the code, maintained it in production. The thinking and upside of this was that quality wasn’t an externality. If your code sucked, then you had a bad experience with ops, and you were motivated to fix it. Where that could fail is when managers didn’t have skin in the game- it’s important that managers have to feel some of the ops pain personally to prevent psychopaths from destroying a team just so they could look good schedule-wise. Note also that “tech debt” either became an ops problem, which self-corrected as described above, or it became an agility problem, where it took forever to make changes to code. DevOps as I saw it didn’t have a good answer to the latter. We had lots of different teams for different projects; having one team know everything about everything wasn’t a thing. Perhaps the problem is that DevOps is not well suited to smaller or less differentiated organizations.
  • x86x87 3y ago
    Imho this is as close as you get to what the intent of devops was originally.
• deleted 3y ago
  [deleted]
• highwaylights 3y ago
  This isn’t complicated. The charlatans moved from Agile to DevOps, taking another reasonable idea and turning it into the exact problem it was supposed to solve, and the cycle repeats (because they need to eat too). Much like with the agile debacle, the best thing to do is continue working and try to avoid/minimise the harm they can do until they find the next idea to latch onto. Incidentally, this video popped into my head as I always think of it when this comes up: https://m.youtube.com/watch?v=0qFGs-SIWB4&pp=ygUeRGVlcGFrIGNob3ByYSBMZW9uYXJkIE1sb2Rpbm93 https://m.youtube.com/watch?v=0qFGs-SIWB4&pp=ygUeRGVlcGFrIGN...
• TheRealDunkirk 3y ago
  But who is going to manage the year-and-a-half long, Excel-checklist-driven process of getting application changes reviewed and approved, and the infrastructure provisioned for each environment? And who is going to keep track of the permissions when forcing everyone to use middleware to access the databases and jump servers to access the machines? My god, won't anyone think of the middle managers!?
• ldehaan 3y ago
  [dead]
• softwaredoug 3y ago
  The core of "Dev Ops" ought to be social incentives. You should eat where you sh*t. If you deploy something, it breaks, you need to be on call to handle it. If you're on call, handling incidents, you need to be also building stuff.
• deleted 3y ago
  [deleted]
• oofnik 3y ago
  DevOps was supposed to be about accelerating the software development lifecycle. In my current role as a "DevOps Engineer", a significant portion of my time is spent being a roadblock. This is mostly due to misaligned incentives and incompetent management. It's debilitating for both the devs and myself. If platform engineering can fix it, fine, but I'm not holding my breath.
• anotherhue 3y ago
  > A great developer platform meets engineers where they are. Many of them are fresh out of code bootcamps. I'm uninterested in subsidising poor education an laziness. Ops is hard. Computers are hard.
  • asynchronous 3y ago
    Ops is hard as heck especially if you have any sort of odd networking requirements. I know dozens of devs that don’t know one thing about networking and would struggle in that field.
    • anotherhue 3y ago
      The author is correct of course, which is why the big techs have built a way of turning junior devs into productivity. The cost of those platforms (the borgs, the chubbys, the map reduces) is stratospheric compared to what a startup should be focusing on. If you want to go somewhere new (as most startups do) you're not going to find a paved road to bring you there. You can ignore architecture but it won't ignore you.
• tabtab 3y ago
  No methodology can make up for bad management. Co's are quick to fire regular employees, but are slow to fire bad managers (or move them to a bitter fit). And KISS & YAGNI should apply to methodologies just as much as software systems. Don't turn simple things into rocket science just to fit some management fad. Have "process meetings" with everyone, collect feedback and discuss areas to improve. Be an ear first, mouth second.
• hosh 3y ago
  Yup, being someone who has done both dev and ops, that is what I see too. In my current current company, the operations group’s name changed from “techops” to “platform”, … but there is no “platform” or roadmap towards that, and it is reflected in how people still call the group “techops”.
• discussDev 3y ago
  There's a bunch of things I agree or don't agree with in this but I'll put my unpopular opinion that I see as an issue even here... The point of devops is to build a consistent platform across all environments programmatically. If you have engrs using a tarball or setting it up some other way then you are doing it wrong and of course you have issues. Also all of your in house projects should use a similar technology stack that you are 100% experts on, half the time someone wants some node instead of php-fpm or something its because they just don't want to learn the other platform... or they are bored and want to add tech just to try new things, don't let them. Build consistent application stacks, deploy them consistently... way fewer issues, way better knowledge transfer, way more maintainable, way cheaper. Hire devs that even want to use the same IDE, same operating systems, again everyone has the same issues and they can be better resolved. Edit: This allows the devs to focus on the business problems and solve them within the stack instead of spending time fighting the technology, it works great.
• lamontcg 3y ago
  Really what happened is that Devs revolted against carrying pagers for their own crappy services, and they reinvented ops teams again. And Amazon has a vested interest in making The Cloud as complicated as possible so that you wind up getting suckered into vendor-lock-in and can never migrate away. Combined with all the overly complicated crap surrounding concepts like CI/CD, K8s and the whole CNCF landscape with a couple thousand different vendors trying like hell to get their tentacles into your organization.
• Tainnor 3y ago
  I'm not sure I even understand the article or what point it is making, as it seems somewhat removed from my own experiences. The expletives also don't help to present a nuanced argument. I'm not in a big corp, we are a rather small team, but we do all of our operations ourselves and it doesn't appear to particularly suck. Sure, the more the complexity of a system grows, the more you might have to split up responsibilities so that not everyone does everything... but not every business is giga-scale and needs kubernetes, at least not initially. Then again, I also very rarely have deadlines and don't work at nights...
• danwee 3y ago
  Perhaps I'm a bit cynic here but this whole mess, turmoil, and hype is so far playing in our advantage (as "engineers"). If any, the real ones that are losing are the companies themselves (because we could work much much efficiently but we don't because we cannot even agree on what DevOps means, or if we actually need DevOps or not). This regardless of the latest trend of layoffs we have seen (I'm considering the whole planet, not just SV). Don't get me wrong, I'm all for efficiency and the like, but if I were a mere "programmer" or "sysadmin" or "webmaster" I don't think I would earn as much as I do now that I'm a "platform/software engineer". Words matter. If you're called an "engineer" (regardless of whether you have a degree in engineering or not), then business people will think something more of you than if you're just a "programmer". Same for "platform" and "devops". Same for "data scientist" and "statistician". You think knowing K8s justifies the $150K/year paycheck in your startup that has only 20 "engineers" and merely 5 services on production? C'mon. But here we are, and I'm more than happy to get those $150K/year for applying K8s and Terraform when it's not actually needed at all. Posts like this one only reminds us that we live in a time where people is making money out of hype (scrum masters anyone?)... but as long as I'm making money as well, I'm happy to join the conversation and play the "let's fix this!" pantomime.
• g9yuayon 3y ago
  Maybe I'm biased, but my experience in Netflix was so good that I never felt the need for a DevOps team. Netflix engineering team did three things right: 1. Any task is an API away and everything except getting a Cassandra keyspace is self-serving. And getting Cassandra access is a matter of a 2-minute conversation. 2. There is always a UI for casual tasks, such as setting up a service cluster or CI/CD pipeline. This is crucial as engineers do not have to learn any non-transferrable shit. The UI will guide them to complete their tasks. 3. Powerful support of observability. Clients, servers, OS, networking... All kinds of metrics are just a finger snap away. In addition, adding metrics is dead easy. If we need something that is below the service level, such as SNMP, we just dropped an email to the four-person observability team, and the metrics will show up in a matter of day before I even had time to finish my own coding. Heck, we had full support of OLAP query on millions of time series with full support of boolean search by any tag (even metric name is a tag) back in 2011, when most people thought Graphite was the best thing since toilet. God! I miss so much the Netflix days when everything was a non-event. Deploying a predictive autoscaling service to any application in production? Non-event. Getting Druid to production readiness in less than a week? Non-event. Having user-controlled fault injection via APIs? Non-event. Setting up Mesos and Zookeeper and opening up Zookeeper for the entire company to use in production within a week? Non-event. Creating a company-wide crypto service and using it to swap AWS keys automatically for all the Netflix services in production and in test? Non-event. Generating an event with a new schema without talking to anyone and see the event show up in a brand new Hive table under 15 minutes. Non-event, and it was back in 2010. Setting up eBPF traces and access all the pretty and dynamically updated charts for any server via an API? Non-event. I don't know what more I could possibly ask for.
• f2hex 3y ago
  Oh yes... the author knows about DevOps... especially when he is writing "We have spent the past 15 years working as DevOps engineers." DevOps is not a role is a practice. That article is really bullshit.