9 ms·

What twenty years of DevOps has failed to do

86 points by mooreds 10mo ago

blutoot 10mo ago
I can't wait for indie developers to build super-agents that commoditize providers like Honeycomb.io and more importantly clone all their features and offer them up for free as OSS.
- verdverm 10mo ago
  Sounds like you don't know what a nightmare of version compat and bespokeness ops/obv is. This is going to be one of the harder things for LLMs to do because everyone is running on some snowflake held together with duct tape
  - blutoot 10mo ago
    Fair point - my statement is more about stealing market for simpler integrations by undercutting them on price. And I don't want to trivialize the reality of enterprise platforms where bespoke connectors rule. I have dealt with migrations of platforms that are business critical and managing version compatibility and ensuring none of the integrations regressed was par for the course. I am not even saying that that makes me qualified to replicate Honeycomb.io. But I do think someone with a deep technical background in building observability platforms armed with Claude Code or Codex and armed with the right set of MCP's and all the necessary tooling should be able to build a clone of Honeycomb.uio. Maybe it won't be a fast turnaround like a typical vibe-coded project but even if it is a month-long project to even get to 60% feature parity. these vendors will have to sit up and pay attention.
    - verdverm 10mo ago
      > And I don't want to trivialize... as you immediately trivialize something it seems you know very little about MCPs are outdated btw, it's bad to attach a bunch of MCPs in with your messages, pollutes the context. If you don't do this, you can build agents that are better than copilot/codex on gemini-3-flash. Claude Code is probably the leader here, but still definitely not capable of what you it is
- hahahahhaah 10mo ago
  I assume then you are retired or not a programmer as you are wishing for the last bastions of comoanies that pay programmers to melt with the ice sheets, leaving the desert of no paid coding work.
mjr00 10mo ago
> most orgs are used to responding to a daytime alert by calling out, “Who just shipped that change?” assuming that whoever merged the diff surely understands how it works and can fix it post-haste. What happens when nobody wrote the code you just deployed, and nobody really understands it? I assume the first time this happens at any given company will be the moment they realize fully autonomous code changes made on production systems by agents is a terrible idea and every change needs a human to take responsibility for and ownership of it, even if the changes were written by an LLM.
- hippo22 10mo ago
  What happens if the person who wrote the code went on vacation? What happens if the code is many years old and no current team member has touched the code? Understanding code you didn't personally write is part of the job.
  - solid_fuel 10mo ago
    I agree that understanding legacy code and code by other people is part of the job, but I don't see how these points are related. > What happens if the person who wrote the code went on vacation? They get yelled at, because shipping code at 5 pm on Friday and then leaving for vacation is typically considered a "dick move". > What happens if the code is many years old and no current team member has touched the code? Then the issue probably isn't caused by a recent deployment?
- blutoot 10mo ago
  I think the opposite will happen - leadership will forego this attitude of "reverse course on the first outage". Teams will figure out how to mitigate such situations in future without sacrificing the potential upside of "fully autonomous code changes made on production systems" (e.g invest more in a production-like env for test coverage). Software engineering purists have to get out of some of these religious beliefs
  - verdverm 10mo ago
    > Software engineering purists have to get out of some of these religious belief To me, the Claude superfans like yourself are the religious, like how you run around poffering unsubstantiated claims like this and believe in / anthropomorphize way too much. Is it because Anthrop'ic is an abbreviation of Anthropomorphic?
politelemon 10mo ago
If your developers weren't looking at dashboards before, they won't use a chat interface to interrogate it either. That doesn't really bring it to them any more than their existing capabilities. There's also a worrying underlying assumption being made here that the answers your LLM will give you are accurate and trustworthy.
- amtamt 10mo ago
  > There's also a worrying underlying assumption being made here that the answers your LLM will give you are accurate and trustworthy. I first hand saw in, AWS devDays, an AI giving SIWINCH as "root-cause" of Apache error in a containerized process is in EKS for a backend FCGI process connection error. It has been extremely hard since that demo to trust any AI for system level debugging.
  - temp0826 10mo ago
    If we were smart we'd use AI to grok a system in order to help us reduce its complexity. I don't think we're anywhere close to even being able to provide all the necessary context to solve problems like this.
  - verdverm 10mo ago
    (1) when was that? If it was less < 6months ago, the current gen of models is noticeably better (2) AWS is not a leader, if even a contender, in the AI space. I would not evaluate the potential based on a demo they produced
- verdverm 10mo ago
  My underlying assumption is that this is a content marketing piece to show managers / investors that "we are doing/thinking something in ai as a company"
jhawk28 10mo ago
DevOps is dead because it's run by a bunch of ops people who don't know how to do dev and a bunch of dev people who don't know how to do ops. The only tooling problem is that a bunch of companies created "DevOps tools" that then get dictated to use: K8s, terraform, etc. The only way this works is if you build the application to fit within those frameworks. Writing an indexer that is massively parallel and is mainly constrained by CPU/Memory. Instead, you have devs building something that gets thrown over the fence to a devops team that then containerizes it and throw it on K8s. What happens if the application requires lots of IOPS or network bandwidth? K8s doesn't schedule applications that way. "Oh you can customize the scheduler to take that into account". 2 years later, it's still not "customized" because they are ops people who don't know how to code. If you do customize it, the API is going to change in a few months which will break when you upgrade.
- blutoot 10mo ago
  [flagged]
  - gmane 10mo ago
    Spoken like someone who has never had to deal with business critical production environments.
    - blutoot 10mo ago
      [flagged]
  - pryelluw 10mo ago
    It’s like saying that in a post-Viagra world there shouldn’t be men who have trouble getting laid.
    - blutoot 10mo ago
      Don't want to get too deep into your analogy. I was addressing the "DevOps cannot code" part. To me it is a leadership failure if a DevOps team is still afraid of tackling bigger challenges (like the example given by the OP). That, of course, depends on whether DevOps teams will exist in the long run.
mosura 10mo ago
It failed because there is an ongoing denial that development and operations are two distinct skillsets. If you think 10x devs are unicorns consider how much harder it is to get someone 10x at the intersection of both domains. (Personally I have never met one). You are far better off with people that can work together across the bridge, but that requires actual mutual trust and respect, and we’re not able to do that.
- tsss 10mo ago
  You don't need 10x developers. You just need to avoid the 1/10 multiplier of pitting separate development and operations teams against each other.
- vee-kay 10mo ago
  From someone who has managed both Developmentals team and Operations team for decades.. trust me, they are different beasts and have to be handled/tackled differently. Expecting Devs or Ops to do both types of work, is usually asking for trouble, unless the organization is geared up from the ground up for such seamless work. It is more of a corporate problem, rather than a team working style or work expectations & behavior problem. The same goes for Agile vs Waterfall. Agile works well if the organization is inherently (or overhauled to be) agile, otherwise it doesn't.
  - SkiFire13 10mo ago
    > Expecting Devs or Ops to do both types of work, is usually asking for trouble, unless the organization is geared up from the ground up for such seamless work. Could you expand on this? How would an organization be geared up for this?
    - vee-kay 10mo ago
      Best example is the largest eCommerce conglomerate in the world: Amazon. In the early 2000s, Google, Amazon and few other companies, were trying to crack the conundrum of APIs (Application Programmable Interfaces), i.e., do web-services. Amazon cracked that conundrum the best and fastest way (and ultimately that's its AWS (Amazon Web Service) cloud platform rules the corporate world). But how did Amazon do that IT innovation (beating other innovative IT services companies like Google), when it was merely an e-commerce and related site/company till then? It's because Amazon did the unthinkable. They overhauled how their company worked. Crucially, Amazon’s engineering teams were instructed in a 2002 memo - by Jeff Bezos, no less - to take an API-first approach: 1. Teams will expose their data and functionality through service interfaces. 2. Teams must communicate with each other through these interfaces. 3. No other form of interprocess communication is allowed. 4. Teams must plan for all service interfaces to be exposed to outside developers. This mandate that every department in the company must communicate internally only using webservises was pivotal to the company's IT-focused transformation. So if HR needed to share some payroll related data to Finance team, it needed to do that communication via APIs, instead of traditional way of attaching it via email or sharing over the shared network drive. This kind of forced, rigorous inter- & intra- communications made the Amazon teams to encounter and resolve every type of issues and concerns that could be faces with APIs/we services and related IT technology. And thus, Amazon was ready with such incredible new innovative, robust and scalable functionality ahead of every other company in the world. Today, AWS is THE cloud platform of choice, and it helps drive most of the biggest websites and platforms of the internet and it has world's biggest/richest companies as its customers. https://gatheringclouds.substack.com/p/the-rise-of-amazon-web-services https://gatheringclouds.substack.com/p/the-rise-of-amazon-we... I feel that simply forcing some Developmental teams to adopt DevOps or Agile in a company doesn't work, if the rest of the company doesn't support DevOps or Agile to the extent needed on a daily basis. Only such deep overhaul can ensure these sorts of innovative best practices can not only survive, but also thrive in the company. And that's the only way the benefits of such radical changes can be felt where it's needed most: the customer experience and the revenue books.
gardenhedge 10mo ago
In my company, instead of relying on an ops team.. we rely on a devops team.
zug_zug 10mo ago
"I think the entire DevOps movement was a mighty, ... it failed." I'm so sick of this nonsense. "Devops" isn't failing, isn't an issue, you can rename it whatever you want, but throughout my career the devops engineers (the ones you don't skimp on) are the best, highest paid professionals at the company. I don't know why I keep reading these completely crazy think-pieces hemming and hawing about a system (having a few engineers who master performance/backups/deployments/oncall/retros) that seems to be wildly successful. It would be nice if more engineers understood under-the-hood, but most companies choose not to exclusively hire at that caliber.
- verdverm 10mo ago
  For sure, I just turned down a gig because the company saw devops as an afterthought, not as something they would invest in. They wanted me to come in and "fix some issues quick" on a short-term contract. What they really need is 1-2 FTE ops people who think about their problems every day. If you are pushing past 3-4 devs to 10 and you have no intention of hiring a FTE ops person, you are not doing it right and shall reap what you have sown before long
blutoot 10mo ago
My message to the CTO of Honeycomb.io (who apparently wrote this post): please avoid getting philosophical and controversial to gin up curiosity about your AI platform. If you want to highlight the benefits of your platform then do so earnestly and objectively. Please don't mask marketing with an excoriation of a profession that has never been well-defined (or has always been defined to fit into an organization's political landscape for the most part). And you guys (like every other SRE/Ops platform) capitalized on that structural divide and deservedly got rich by selling licenses to these teams. I don't think you can come in now with this holier-than-thou best practice messaging just because platforms like yours have zero moat in this post-CC/Codex world. Hence my vitriol: https://news.ycombinator.com/item?id=46662287 https://news.ycombinator.com/item?id=46662287.
- TacticalCoder 10mo ago
  > id getting philosophical and controversial to gin up curiosity about your AI platform Also: please could he please avoid doing it by illustrating his non-sense with graphs that are both childish and non-sensical?
  - maccard 10mo ago
    The CTO is a she.
zero_shift 10mo ago
Because the idea you can have all aspects of maintaining a complex piece of technology, maintained by a single cross-skilled team of interchangeable cogs, is utopian and unworkable past any reasonable level of scale DevOps, shift left, full stack dev, all reminds me of the Futurama episode where Hermes Conrad successfully reorgs the slave camp he's sent to, so that all physical labour is done by a single Australian man Speaking darker, there is a kind of - well, perhaps not misanthropy, but certainly a not-so-well-meaning dismissiveness, to the "silo breaking" philosophy that looks at complex fields and says "well these should all just be lumped together as one thing, the important stuff is simple, I don't know why you're making all these siloes, man" - assuming that ops specialists, sysadmins, programmers, DBAs, frontend devs, mobile devs, data engineers and testers have just invented the breadth and depth and subtleties of their entire fields, only as a way of keeping everybody else out But modern systems are complex, they are only getting more so, and the further you buy into the shift-left everyone-is-everything computer-jobs-are-all-the-same philosophy the harder and harder it will get to find employees who can straddle the exhausting range of knowledge to master
- lll-o-lll 10mo ago
  > the "silo breaking" philosophy that looks at complex fields and says "well these should all just be lumped together as one thing, the important stuff is simple, I don’t think this is the right take. “Silo’s” is an ill-defined term, but let’s look at a couple of the negative aspects. “Lack of communication”, and “Lack of shared understanding” (or different models of the world). I’m going to use a different industry example, as I think it helps think about the problem more abstractly. In the world of biomedical engineering, the types of products you are making require the expertise of two very different groups of people. Engineers and Doctors. A member of either of these groups have an in-group language, and there is an inherent power differential between them. Doctors are more “important” than engineers. But to get anything made, you need the expertise of both. One way to handle this is to keep the engineers and doctors separate and to communicate primarily via documents. The doctor will attempt to detail exactly how a certain component should work. The engineer will attempt to detail the constraints and request clarifications. The problem with this approach is that the engineer cannot speak “doctorese” nor can the doctor speak “engineerese”; and the consequence is a model in each person’s head that differs significantly from the other. There is no shared model; and the real world product suffers as a result. The alternative is to attempt to “break the silos”; force the engineers and doctors to sit with each other, learn each other’s language, and build a shared mental model of what is being created. This creates a far better product; one that is much closer to the “physical reality” it must inhabit. The same is true across all kinds of business groups. If different groups of people are required to collaborate, in order to do something, those people are well served by learning each other’s languages and building a shared mental model. That’s what breaking silos is about. It is not “everyone is the same”, it’s “breaking down the communication barriers”.
alphazard 10mo ago
DevOps only works when the developers are always right. What usually happens is the DevOps team thinks they know best (they are developers too, just not the ones using the tools), and they build a lot of garbage that no one wants to use, often making things more complicated than they were before. Eventually a bureaucrat becomes the manager of the team, and seeks to expand the set of things under DevOps' control. This makes the team a single point of failure for more and more things, while driving more and more developer processes towards mediocrity. Velocity slows, while the DevOps bottlenecks are used as a reason to hire. It's an organizational problem, not a talent or knowledge problem. Allowing a group to hire and grow within an organization, which is not directly accountable for the success of the other parts of the organization that it was intended to support, is creating a cancer, definitionally.
- verdverm 10mo ago
  Don't attribute internal dysfunction and mistakes to an entire field. I've worked in an org where the opposite is true. Blanket statements like these never hold up because they lack nuance and usually are inspired by frustration
  - alphazard 10mo ago
    If an idea or methodology leads to "internal dysfunction" more often than not, than maybe it's a problem with the idea, and not a "lack of nuance".
    - verdverm 10mo ago
      which idea is it that leads to the internal dysfunction? The one where someone has already made their mind up about what does and doesn't work for everyone everywhere?
anonymars 10mo ago
Am I the only one who remembers when DevOps meant "developers are responsible for dealing with the operational part of their software too, so that they don't just throw stuff over the wall for another team to deal with the 3AM pages"? It seems to have become: "we turned ops into coding too, so now the ops team needs to be good at software engineering"
- prmoustache 10mo ago
  I am with you. DevOps is a methodology. DevOps as a role or team name is a fantasy from people who do not understand the methodology. If you want DevOps to work, your Ops must be member of the development team, take part in the sprints, etc. But many company do not want to do that because they want to separate ops and dev budget/accounting and do not want to hire enough people with ops skills.
  - verdverm 10mo ago
    This is not true, you can make it work well either way. It's about people and processes, not about some specific setup or way of grouping people
- vee-kay 10mo ago
  DevOps was (and is) merely an excuse for companies to replace Developers with cheaper Ops resources, and yet expecting better services and better products from them. My personal experience says that the best way is that Ops team shouldn not be repurposed as Developers, rather put the experienced Developers into Production Support (incident management, that's intense Ops, working in shifts and weekends, etc.). And rotate them whenever needed. Over a period of time, you'll invariably see less defects and issues percolating down from the Devs, and then after both sides are stable and working well together with less friction and open tickets, then some more tech savvy Ops members can be rotated into Development teams as rookie devs to help reduce costs a bit (as there'll invariably be some natural attrition among the Devs and Ops, so this gives an alternative career path to the Ops team (who are usually less paid, and more stressed), and pushes the Devs not to become complacent). Such an approach is doable and productive.
skybrian 10mo ago
I don't understand these graphs. Why do the lines go back in time?
- verdverm 10mo ago
  feed"back" loops
bravetraveler 10mo ago
Scratching neck: come on... just one more vendor, bro
GiorgioG 10mo ago
In my experience DevOps has little interest in doing actual DevOps - they just want to run ops. They want to advise (or tell us we’re holding it wrong) but not actually get their hands dirty. On the flip side, devs don’t want to spend a ton of time learning k8s or how to manage servers, cloud services, etc. DevOps is a mess of our own making - embracing K8s created complexity for little gain for nearly all companies.
bmitch3020 10mo ago
DevOps only failed in that so many don't know what it is. DevOps isn't a tool, but there are lots of tools that make it easier to implement. DevOps isn't how management can eliminate half the org and have one person do two roles, specialization is still valuable. DevOps isn't an organization structure, though the wrong org structure can make it fail. DevOps is collaboration. It's getting two distinct roles to better interoperate. The dev team that wants to push features fast. And the ops team that wants stability and uptime. From the management side, if you aren't focused on building teams that work well together, eliminating conflicts, rewarding the team collectively for features and uptime, and giving them the resources to deliver, that's not a DevOps failure, that's a management failure.
- acdha 10mo ago
  I think this is the key insight: most of the problems are related to management decisions so they’re only DevOps failures to the extent that the movement failed to get political pressure to fix those.
verdverm 10mo ago
Yaml is my #1 failure in devops. That so many have resigned themselves to this limit and no longer seek to improve, it's disappointing. Our job is to make things run better and easier, yet so many won't recognize the biggest pains in their own work. Seriously, is text templating an invisibly scoped language really where you think the field has reached maturity?
- firesteelrain 10mo ago
  JSON so much easier in my experience and less prone to error
  - verdverm 10mo ago
    JSON does not have comments, no JSON5 is not the answer either Think bigger, it's not something you are using today. The next config language should have schemas built in and support for modules/imports so we can do sharing/caring. It should look and feel like config languages and interoperate with all of those that we currently use. It will be a single configuration fabric across the SDLC. This exists today for you to try, with CUE I've been cooking up something the last few weeks for those interested, CUE + Dagger https://github.com/hofstadter-io/hof/tree/_next/examples/env https://github.com/hofstadter-io/hof/tree/_next/examples/env
    - lijok 10mo ago
      Like Python?
      - verdverm 10mo ago
        CUE
        lijok 10mo ago
        Why not Python?
mgilroy 10mo ago
I'd argue that it has failed in some organisations. DevOps for me is embedding the operations with the development team. I still have operations specialist, however, they attend the development team stand ups and help articulate the problems to the developers. They may have separate operations standups and meetings to ensure the operations teams know what they are doing and share best practices. Developers learn about the operations side from those that understand it well and the operations experts learn the limitations and needs of the developers. Occasionally I am fortunate to discover someone's that can understand both areas incredibly well. Either way, this results in increased trust and closer working. You don't care about helping some random person on a ticket from a tream you don't know. You do care about the person you work with daily and understand the problems they have. If you can't account for someone spending x% of their time working with a team but for budgetary purposes belonging to a different team then sack your accountants. DevOps,like agile, when done correctly should help to create teams that understand complete systems or areas of a business work more efficiently than having stand alone teams. The other part of the puzzle is to include the QA team too to ensure that the impact of full system, performance and integration tests are understood by all and that both everyone understands how their changes impact everything else. Having the dev team build code that makes the test and ops teams life easier benefits everyone. Having the ops team provide solutions that support test and dev helps everyone. Having test teams build system that work best with the Dev and ops teams helps everyone. Agile development should enable teams to work at a higher level of performance by granting them the agency to make the right decisions at the right time to deliver a better product by building what is needed in the correct timeline. DevOps and agile fail where companies try to follow waterfall models whilst claiming agile processes. The goal with all these business and operating models is to improve efficiency. When that isn't happening then either you aren't applying the model correctly or you need to change the model.
browningstreet 10mo ago
I miss good ol’ classical sys admin.
VerifiedReports 10mo ago
Has this buzzword even been around 20 years?
maerF0x0 10mo ago
I'm surprised no one has commented that open loops plus closed loops is more or less how velcro works, and that stuff sticks.
0xbadcafebee 10mo ago
As a movement, DevOps failed a long time ago. Once the word completely lost its meaning, it was impossible to educate anyone about it. But as a business practice (which is mostly what it is), it's still a viable option that any business can implement. It just takes the right people rising into leadership positions to enact it.
BobbyTables2 10mo ago
Like other fads, it has certainly failed to go out of existence as soon as it should have.
kittikitti 10mo ago
The vast majority of people who called themselves DevOps were opportunist corporatists looking to get a promotion. It became less Ops as in operations and more Ops as in opponent. These people were haphazardly given the keys to production and I was often bullied by their ability to control it. Management needs to know if my application was running? They asked DevOps who often came back with lies in order to fit their own career objectives. I apologize if my words were sharp because many DevOps engineers were not mean to me. Perhaps I just had bad luck to deal with ignorant gatekeepers to production. You already know if my opinion doesn't apply to you.
iwontberude 10mo ago
You made a graph with T being the x axis and then had lines which go backwards in time. I closed the window at this point.
pjmlp 10mo ago
It failed because for management and HR, DevOps means ops, for them it is the new buzzword for systems integration, integration engineers, sysadmin,.. I have been foolish enough to accept a few project proposals with DevOps role, which in the end meant ops work dealing with VMs, networking and the like.
antod 10mo ago
I don't know of any other term in tech that people experience in so many different often contradictory ways that causes people to talk past each other because they're all talking about different things or been places that work so differently.
- hahahahhaah 10mo ago
  Agile is like this too.
  - antod 10mo ago
    Yeah I thought of that one, but still reckon DevOps goes well past it.
omnifischer 10mo ago
From the article: > What the devs care about is the ability to understand the product experience from the perspective of each customer. In practice, this can mean any combination or permutation of agent, user, mobile device type, laptop, desktop, point of sale device, and so on Really? Any permutation? Most (arse hole) devs - Import world - as it works on their latest 1TB machine or macOS studio - always on the latest iPhone or pixel - Add 100 tracking that works on their own machine - POS device? They should ask some of devs to go and work on their canteens that have POS
hahahahhaah 10mo ago
I am lucky that the article is alien to me. I work somewhere we do devops. I write code and tests etc. but I would unlikely go a day without observing something in prod either by telemetry, dogfooding or getting paged. It is a really cool way to work understanding the whole system (within the team remit of course). And vaguely understanding neighbouring teams. To the comments dev and ops are different. They are! I think the magic is massive platform team support too. I am not troubleshooting why splunk indexes aren't indexing for example.
deleted 10mo ago
[deleted]
phendrenad2 10mo ago
DevOps failed because management never really understood that it needed to be a dedicated team, and that it shouldn't just be a little vassal state of the ops team, nor is it just "backend devs who know ops" who keep getting pulled away from devops work to do backend work. IF DevOps had been allowed to thrive, we could have eventually reached the holy grail: A Heroku-like environment that developers can use to run their apps. Instead, we stopped short in the no-mans land of k8s, helm, AWS lock-in, terraform, hashicorp, red hat, azure, not to mention tons of spreadsheets owned by the ops team that developers must beg and plead to get changed.

86 points by mooreds 10mo ago

I can't wait for indie developers to build super-agents that commoditize providers like Honeycomb.io and more importantly clone all their features and offer them up for free as OSS.

verdverm 10mo ago
Sounds like you don't know what a nightmare of version compat and bespokeness ops/obv is. This is going to be one of the harder things for LLMs to do because everyone is running on some snowflake held together with duct tape
- blutoot 10mo ago
  Fair point - my statement is more about stealing market for simpler integrations by undercutting them on price. And I don't want to trivialize the reality of enterprise platforms where bespoke connectors rule. I have dealt with migrations of platforms that are business critical and managing version compatibility and ensuring none of the integrations regressed was par for the course. I am not even saying that that makes me qualified to replicate Honeycomb.io. But I do think someone with a deep technical background in building observability platforms armed with Claude Code or Codex and armed with the right set of MCP's and all the necessary tooling should be able to build a clone of Honeycomb.uio. Maybe it won't be a fast turnaround like a typical vibe-coded project but even if it is a month-long project to even get to 60% feature parity. these vendors will have to sit up and pay attention.
  - verdverm 10mo ago
    > And I don't want to trivialize... as you immediately trivialize something it seems you know very little about MCPs are outdated btw, it's bad to attach a bunch of MCPs in with your messages, pollutes the context. If you don't do this, you can build agents that are better than copilot/codex on gemini-3-flash. Claude Code is probably the leader here, but still definitely not capable of what you it is
hahahahhaah 10mo ago
I assume then you are retired or not a programmer as you are wishing for the last bastions of comoanies that pay programmers to melt with the ice sheets, leaving the desert of no paid coding work.

mjr00 10mo ago

> most orgs are used to responding to a daytime alert by calling out, “Who just shipped that change?” assuming that whoever merged the diff surely understands how it works and can fix it post-haste. What happens when nobody wrote the code you just deployed, and nobody really understands it? I assume the first time this happens at any given company will be the moment they realize fully autonomous code changes made on production systems by agents is a terrible idea and every change needs a human to take responsibility for and ownership of it, even if the changes were written by an LLM.

hippo22 10mo ago
What happens if the person who wrote the code went on vacation? What happens if the code is many years old and no current team member has touched the code? Understanding code you didn't personally write is part of the job.
- solid_fuel 10mo ago
  I agree that understanding legacy code and code by other people is part of the job, but I don't see how these points are related. > What happens if the person who wrote the code went on vacation? They get yelled at, because shipping code at 5 pm on Friday and then leaving for vacation is typically considered a "dick move". > What happens if the code is many years old and no current team member has touched the code? Then the issue probably isn't caused by a recent deployment?
blutoot 10mo ago
I think the opposite will happen - leadership will forego this attitude of "reverse course on the first outage". Teams will figure out how to mitigate such situations in future without sacrificing the potential upside of "fully autonomous code changes made on production systems" (e.g invest more in a production-like env for test coverage). Software engineering purists have to get out of some of these religious beliefs
- verdverm 10mo ago
  > Software engineering purists have to get out of some of these religious belief To me, the Claude superfans like yourself are the religious, like how you run around poffering unsubstantiated claims like this and believe in / anthropomorphize way too much. Is it because Anthrop'ic is an abbreviation of Anthropomorphic?

politelemon 10mo ago

If your developers weren't looking at dashboards before, they won't use a chat interface to interrogate it either. That doesn't really bring it to them any more than their existing capabilities. There's also a worrying underlying assumption being made here that the answers your LLM will give you are accurate and trustworthy.

amtamt 10mo ago
> There's also a worrying underlying assumption being made here that the answers your LLM will give you are accurate and trustworthy. I first hand saw in, AWS devDays, an AI giving SIWINCH as "root-cause" of Apache error in a containerized process is in EKS for a backend FCGI process connection error. It has been extremely hard since that demo to trust any AI for system level debugging.
- temp0826 10mo ago
  If we were smart we'd use AI to grok a system in order to help us reduce its complexity. I don't think we're anywhere close to even being able to provide all the necessary context to solve problems like this.
- verdverm 10mo ago
  (1) when was that? If it was less < 6months ago, the current gen of models is noticeably better (2) AWS is not a leader, if even a contender, in the AI space. I would not evaluate the potential based on a demo they produced
verdverm 10mo ago
My underlying assumption is that this is a content marketing piece to show managers / investors that "we are doing/thinking something in ai as a company"

jhawk28 10mo ago

DevOps is dead because it's run by a bunch of ops people who don't know how to do dev and a bunch of dev people who don't know how to do ops. The only tooling problem is that a bunch of companies created "DevOps tools" that then get dictated to use: K8s, terraform, etc. The only way this works is if you build the application to fit within those frameworks. Writing an indexer that is massively parallel and is mainly constrained by CPU/Memory. Instead, you have devs building something that gets thrown over the fence to a devops team that then containerizes it and throw it on K8s. What happens if the application requires lots of IOPS or network bandwidth? K8s doesn't schedule applications that way. "Oh you can customize the scheduler to take that into account". 2 years later, it's still not "customized" because they are ops people who don't know how to code. If you do customize it, the API is going to change in a few months which will break when you upgrade.

blutoot 10mo ago
[flagged]
- gmane 10mo ago
  Spoken like someone who has never had to deal with business critical production environments.
  - blutoot 10mo ago
    [flagged]
- pryelluw 10mo ago
  It’s like saying that in a post-Viagra world there shouldn’t be men who have trouble getting laid.
  - blutoot 10mo ago
    Don't want to get too deep into your analogy. I was addressing the "DevOps cannot code" part. To me it is a leadership failure if a DevOps team is still afraid of tackling bigger challenges (like the example given by the OP). That, of course, depends on whether DevOps teams will exist in the long run.

mosura 10mo ago

It failed because there is an ongoing denial that development and operations are two distinct skillsets. If you think 10x devs are unicorns consider how much harder it is to get someone 10x at the intersection of both domains. (Personally I have never met one). You are far better off with people that can work together across the bridge, but that requires actual mutual trust and respect, and we’re not able to do that.

tsss 10mo ago
You don't need 10x developers. You just need to avoid the 1/10 multiplier of pitting separate development and operations teams against each other.
vee-kay 10mo ago
From someone who has managed both Developmentals team and Operations team for decades.. trust me, they are different beasts and have to be handled/tackled differently. Expecting Devs or Ops to do both types of work, is usually asking for trouble, unless the organization is geared up from the ground up for such seamless work. It is more of a corporate problem, rather than a team working style or work expectations & behavior problem. The same goes for Agile vs Waterfall. Agile works well if the organization is inherently (or overhauled to be) agile, otherwise it doesn't.
- SkiFire13 10mo ago
  > Expecting Devs or Ops to do both types of work, is usually asking for trouble, unless the organization is geared up from the ground up for such seamless work. Could you expand on this? How would an organization be geared up for this?
  - vee-kay 10mo ago
    Best example is the largest eCommerce conglomerate in the world: Amazon. In the early 2000s, Google, Amazon and few other companies, were trying to crack the conundrum of APIs (Application Programmable Interfaces), i.e., do web-services. Amazon cracked that conundrum the best and fastest way (and ultimately that's its AWS (Amazon Web Service) cloud platform rules the corporate world). But how did Amazon do that IT innovation (beating other innovative IT services companies like Google), when it was merely an e-commerce and related site/company till then? It's because Amazon did the unthinkable. They overhauled how their company worked. Crucially, Amazon’s engineering teams were instructed in a 2002 memo - by Jeff Bezos, no less - to take an API-first approach: 1. Teams will expose their data and functionality through service interfaces. 2. Teams must communicate with each other through these interfaces. 3. No other form of interprocess communication is allowed. 4. Teams must plan for all service interfaces to be exposed to outside developers. This mandate that every department in the company must communicate internally only using webservises was pivotal to the company's IT-focused transformation. So if HR needed to share some payroll related data to Finance team, it needed to do that communication via APIs, instead of traditional way of attaching it via email or sharing over the shared network drive. This kind of forced, rigorous inter- & intra- communications made the Amazon teams to encounter and resolve every type of issues and concerns that could be faces with APIs/we services and related IT technology. And thus, Amazon was ready with such incredible new innovative, robust and scalable functionality ahead of every other company in the world. Today, AWS is THE cloud platform of choice, and it helps drive most of the biggest websites and platforms of the internet and it has world's biggest/richest companies as its customers. https://gatheringclouds.substack.com/p/the-rise-of-amazon-web-services https://gatheringclouds.substack.com/p/the-rise-of-amazon-we... I feel that simply forcing some Developmental teams to adopt DevOps or Agile in a company doesn't work, if the rest of the company doesn't support DevOps or Agile to the extent needed on a daily basis. Only such deep overhaul can ensure these sorts of innovative best practices can not only survive, but also thrive in the company. And that's the only way the benefits of such radical changes can be felt where it's needed most: the customer experience and the revenue books.

gardenhedge 10mo ago

In my company, instead of relying on an ops team.. we rely on a devops team.

zug_zug 10mo ago

"I think the entire DevOps movement was a mighty, ... it failed." I'm so sick of this nonsense. "Devops" isn't failing, isn't an issue, you can rename it whatever you want, but throughout my career the devops engineers (the ones you don't skimp on) are the best, highest paid professionals at the company. I don't know why I keep reading these completely crazy think-pieces hemming and hawing about a system (having a few engineers who master performance/backups/deployments/oncall/retros) that seems to be wildly successful. It would be nice if more engineers understood under-the-hood, but most companies choose not to exclusively hire at that caliber.

verdverm 10mo ago
For sure, I just turned down a gig because the company saw devops as an afterthought, not as something they would invest in. They wanted me to come in and "fix some issues quick" on a short-term contract. What they really need is 1-2 FTE ops people who think about their problems every day. If you are pushing past 3-4 devs to 10 and you have no intention of hiring a FTE ops person, you are not doing it right and shall reap what you have sown before long

blutoot 10mo ago

My message to the CTO of Honeycomb.io (who apparently wrote this post): please avoid getting philosophical and controversial to gin up curiosity about your AI platform. If you want to highlight the benefits of your platform then do so earnestly and objectively. Please don't mask marketing with an excoriation of a profession that has never been well-defined (or has always been defined to fit into an organization's political landscape for the most part). And you guys (like every other SRE/Ops platform) capitalized on that structural divide and deservedly got rich by selling licenses to these teams. I don't think you can come in now with this holier-than-thou best practice messaging just because platforms like yours have zero moat in this post-CC/Codex world. Hence my vitriol: https://news.ycombinator.com/item?id=46662287 https://news.ycombinator.com/item?id=46662287.

TacticalCoder 10mo ago
> id getting philosophical and controversial to gin up curiosity about your AI platform Also: please could he please avoid doing it by illustrating his non-sense with graphs that are both childish and non-sensical?
- maccard 10mo ago
  The CTO is a she.

zero_shift 10mo ago

Because the idea you can have all aspects of maintaining a complex piece of technology, maintained by a single cross-skilled team of interchangeable cogs, is utopian and unworkable past any reasonable level of scale DevOps, shift left, full stack dev, all reminds me of the Futurama episode where Hermes Conrad successfully reorgs the slave camp he's sent to, so that all physical labour is done by a single Australian man Speaking darker, there is a kind of - well, perhaps not misanthropy, but certainly a not-so-well-meaning dismissiveness, to the "silo breaking" philosophy that looks at complex fields and says "well these should all just be lumped together as one thing, the important stuff is simple, I don't know why you're making all these siloes, man" - assuming that ops specialists, sysadmins, programmers, DBAs, frontend devs, mobile devs, data engineers and testers have just invented the breadth and depth and subtleties of their entire fields, only as a way of keeping everybody else out But modern systems are complex, they are only getting more so, and the further you buy into the shift-left everyone-is-everything computer-jobs-are-all-the-same philosophy the harder and harder it will get to find employees who can straddle the exhausting range of knowledge to master

lll-o-lll 10mo ago
> the "silo breaking" philosophy that looks at complex fields and says "well these should all just be lumped together as one thing, the important stuff is simple, I don’t think this is the right take. “Silo’s” is an ill-defined term, but let’s look at a couple of the negative aspects. “Lack of communication”, and “Lack of shared understanding” (or different models of the world). I’m going to use a different industry example, as I think it helps think about the problem more abstractly. In the world of biomedical engineering, the types of products you are making require the expertise of two very different groups of people. Engineers and Doctors. A member of either of these groups have an in-group language, and there is an inherent power differential between them. Doctors are more “important” than engineers. But to get anything made, you need the expertise of both. One way to handle this is to keep the engineers and doctors separate and to communicate primarily via documents. The doctor will attempt to detail exactly how a certain component should work. The engineer will attempt to detail the constraints and request clarifications. The problem with this approach is that the engineer cannot speak “doctorese” nor can the doctor speak “engineerese”; and the consequence is a model in each person’s head that differs significantly from the other. There is no shared model; and the real world product suffers as a result. The alternative is to attempt to “break the silos”; force the engineers and doctors to sit with each other, learn each other’s language, and build a shared mental model of what is being created. This creates a far better product; one that is much closer to the “physical reality” it must inhabit. The same is true across all kinds of business groups. If different groups of people are required to collaborate, in order to do something, those people are well served by learning each other’s languages and building a shared mental model. That’s what breaking silos is about. It is not “everyone is the same”, it’s “breaking down the communication barriers”.

alphazard 10mo ago

DevOps only works when the developers are always right. What usually happens is the DevOps team thinks they know best (they are developers too, just not the ones using the tools), and they build a lot of garbage that no one wants to use, often making things more complicated than they were before. Eventually a bureaucrat becomes the manager of the team, and seeks to expand the set of things under DevOps' control. This makes the team a single point of failure for more and more things, while driving more and more developer processes towards mediocrity. Velocity slows, while the DevOps bottlenecks are used as a reason to hire. It's an organizational problem, not a talent or knowledge problem. Allowing a group to hire and grow within an organization, which is not directly accountable for the success of the other parts of the organization that it was intended to support, is creating a cancer, definitionally.

verdverm 10mo ago
Don't attribute internal dysfunction and mistakes to an entire field. I've worked in an org where the opposite is true. Blanket statements like these never hold up because they lack nuance and usually are inspired by frustration
- alphazard 10mo ago
  If an idea or methodology leads to "internal dysfunction" more often than not, than maybe it's a problem with the idea, and not a "lack of nuance".
  - verdverm 10mo ago
    which idea is it that leads to the internal dysfunction? The one where someone has already made their mind up about what does and doesn't work for everyone everywhere?

anonymars 10mo ago

Am I the only one who remembers when DevOps meant "developers are responsible for dealing with the operational part of their software too, so that they don't just throw stuff over the wall for another team to deal with the 3AM pages"? It seems to have become: "we turned ops into coding too, so now the ops team needs to be good at software engineering"

prmoustache 10mo ago
I am with you. DevOps is a methodology. DevOps as a role or team name is a fantasy from people who do not understand the methodology. If you want DevOps to work, your Ops must be member of the development team, take part in the sprints, etc. But many company do not want to do that because they want to separate ops and dev budget/accounting and do not want to hire enough people with ops skills.
- verdverm 10mo ago
  This is not true, you can make it work well either way. It's about people and processes, not about some specific setup or way of grouping people
vee-kay 10mo ago
DevOps was (and is) merely an excuse for companies to replace Developers with cheaper Ops resources, and yet expecting better services and better products from them. My personal experience says that the best way is that Ops team shouldn not be repurposed as Developers, rather put the experienced Developers into Production Support (incident management, that's intense Ops, working in shifts and weekends, etc.). And rotate them whenever needed. Over a period of time, you'll invariably see less defects and issues percolating down from the Devs, and then after both sides are stable and working well together with less friction and open tickets, then some more tech savvy Ops members can be rotated into Development teams as rookie devs to help reduce costs a bit (as there'll invariably be some natural attrition among the Devs and Ops, so this gives an alternative career path to the Ops team (who are usually less paid, and more stressed), and pushes the Devs not to become complacent). Such an approach is doable and productive.

skybrian 10mo ago

I don't understand these graphs. Why do the lines go back in time?

verdverm 10mo ago
feed"back" loops

bravetraveler 10mo ago

Scratching neck: come on... just one more vendor, bro

GiorgioG 10mo ago

In my experience DevOps has little interest in doing actual DevOps - they just want to run ops. They want to advise (or tell us we’re holding it wrong) but not actually get their hands dirty. On the flip side, devs don’t want to spend a ton of time learning k8s or how to manage servers, cloud services, etc. DevOps is a mess of our own making - embracing K8s created complexity for little gain for nearly all companies.

bmitch3020 10mo ago

DevOps only failed in that so many don't know what it is. DevOps isn't a tool, but there are lots of tools that make it easier to implement. DevOps isn't how management can eliminate half the org and have one person do two roles, specialization is still valuable. DevOps isn't an organization structure, though the wrong org structure can make it fail. DevOps is collaboration. It's getting two distinct roles to better interoperate. The dev team that wants to push features fast. And the ops team that wants stability and uptime. From the management side, if you aren't focused on building teams that work well together, eliminating conflicts, rewarding the team collectively for features and uptime, and giving them the resources to deliver, that's not a DevOps failure, that's a management failure.

acdha 10mo ago
I think this is the key insight: most of the problems are related to management decisions so they’re only DevOps failures to the extent that the movement failed to get political pressure to fix those.

verdverm 10mo ago

Yaml is my #1 failure in devops. That so many have resigned themselves to this limit and no longer seek to improve, it's disappointing. Our job is to make things run better and easier, yet so many won't recognize the biggest pains in their own work. Seriously, is text templating an invisibly scoped language really where you think the field has reached maturity?

firesteelrain 10mo ago
JSON so much easier in my experience and less prone to error
- verdverm 10mo ago
  JSON does not have comments, no JSON5 is not the answer either Think bigger, it's not something you are using today. The next config language should have schemas built in and support for modules/imports so we can do sharing/caring. It should look and feel like config languages and interoperate with all of those that we currently use. It will be a single configuration fabric across the SDLC. This exists today for you to try, with CUE I've been cooking up something the last few weeks for those interested, CUE + Dagger https://github.com/hofstadter-io/hof/tree/_next/examples/env https://github.com/hofstadter-io/hof/tree/_next/examples/env
  - lijok 10mo ago
    Like Python?
    - verdverm 10mo ago
      CUE
      - lijok 10mo ago
        Why not Python?

mgilroy 10mo ago

I'd argue that it has failed in some organisations. DevOps for me is embedding the operations with the development team. I still have operations specialist, however, they attend the development team stand ups and help articulate the problems to the developers. They may have separate operations standups and meetings to ensure the operations teams know what they are doing and share best practices. Developers learn about the operations side from those that understand it well and the operations experts learn the limitations and needs of the developers. Occasionally I am fortunate to discover someone's that can understand both areas incredibly well. Either way, this results in increased trust and closer working. You don't care about helping some random person on a ticket from a tream you don't know. You do care about the person you work with daily and understand the problems they have. If you can't account for someone spending x% of their time working with a team but for budgetary purposes belonging to a different team then sack your accountants. DevOps,like agile, when done correctly should help to create teams that understand complete systems or areas of a business work more efficiently than having stand alone teams. The other part of the puzzle is to include the QA team too to ensure that the impact of full system, performance and integration tests are understood by all and that both everyone understands how their changes impact everything else. Having the dev team build code that makes the test and ops teams life easier benefits everyone. Having the ops team provide solutions that support test and dev helps everyone. Having test teams build system that work best with the Dev and ops teams helps everyone. Agile development should enable teams to work at a higher level of performance by granting them the agency to make the right decisions at the right time to deliver a better product by building what is needed in the correct timeline. DevOps and agile fail where companies try to follow waterfall models whilst claiming agile processes. The goal with all these business and operating models is to improve efficiency. When that isn't happening then either you aren't applying the model correctly or you need to change the model.

browningstreet 10mo ago

I miss good ol’ classical sys admin.

VerifiedReports 10mo ago

Has this buzzword even been around 20 years?

maerF0x0 10mo ago

I'm surprised no one has commented that open loops plus closed loops is more or less how velcro works, and that stuff sticks.

0xbadcafebee 10mo ago

As a movement, DevOps failed a long time ago. Once the word completely lost its meaning, it was impossible to educate anyone about it. But as a business practice (which is mostly what it is), it's still a viable option that any business can implement. It just takes the right people rising into leadership positions to enact it.

BobbyTables2 10mo ago

Like other fads, it has certainly failed to go out of existence as soon as it should have.

kittikitti 10mo ago

The vast majority of people who called themselves DevOps were opportunist corporatists looking to get a promotion. It became less Ops as in operations and more Ops as in opponent. These people were haphazardly given the keys to production and I was often bullied by their ability to control it. Management needs to know if my application was running? They asked DevOps who often came back with lies in order to fit their own career objectives. I apologize if my words were sharp because many DevOps engineers were not mean to me. Perhaps I just had bad luck to deal with ignorant gatekeepers to production. You already know if my opinion doesn't apply to you.

iwontberude 10mo ago

You made a graph with T being the x axis and then had lines which go backwards in time. I closed the window at this point.

pjmlp 10mo ago

It failed because for management and HR, DevOps means ops, for them it is the new buzzword for systems integration, integration engineers, sysadmin,.. I have been foolish enough to accept a few project proposals with DevOps role, which in the end meant ops work dealing with VMs, networking and the like.

antod 10mo ago

I don't know of any other term in tech that people experience in so many different often contradictory ways that causes people to talk past each other because they're all talking about different things or been places that work so differently.

hahahahhaah 10mo ago
Agile is like this too.
- antod 10mo ago
  Yeah I thought of that one, but still reckon DevOps goes well past it.

omnifischer 10mo ago

From the article: > What the devs care about is the ability to understand the product experience from the perspective of each customer. In practice, this can mean any combination or permutation of agent, user, mobile device type, laptop, desktop, point of sale device, and so on Really? Any permutation? Most (arse hole) devs - Import world - as it works on their latest 1TB machine or macOS studio - always on the latest iPhone or pixel - Add 100 tracking that works on their own machine - POS device? They should ask some of devs to go and work on their canteens that have POS

hahahahhaah 10mo ago

I am lucky that the article is alien to me. I work somewhere we do devops. I write code and tests etc. but I would unlikely go a day without observing something in prod either by telemetry, dogfooding or getting paged. It is a really cool way to work understanding the whole system (within the team remit of course). And vaguely understanding neighbouring teams. To the comments dev and ops are different. They are! I think the magic is massive platform team support too. I am not troubleshooting why splunk indexes aren't indexing for example.

deleted 10mo ago

[deleted]

phendrenad2 10mo ago

DevOps failed because management never really understood that it needed to be a dedicated team, and that it shouldn't just be a little vassal state of the ops team, nor is it just "backend devs who know ops" who keep getting pulled away from devops work to do backend work. IF DevOps had been allowed to thrive, we could have eventually reached the holy grail: A Heroku-like environment that developers can use to run their apps. Instead, we stopped short in the no-mans land of k8s, helm, AWS lock-in, terraform, hashicorp, red hat, azure, not to mention tons of spreadsheets owned by the ops team that developers must beg and plead to get changed.