10 ms·

It's the end of observability as we know it (and I feel fine)

301 points by gpi 1y ago

techpineapple 1y ago
I feel like the alternate title of this could be “how to 10x your observability costs with this one easy trick”. It didn’t really show a way to get rid of all the graphs, the prompt was “show me why my latency spikes every four hours”. That’s really cool, but in order to generate that prompt you need alerts and graphs. How do you know you’re latency is spiking to generate the prompt? The devil seems to be in the details, but you’re running a whole bunch more compute for anomaly detection and “ Sub-second query performance, unified data storage”, which again sounds like throwing enormous amounts of more money at the problem. I can totally see why this is great for honeycomb though, they’re going to make bank.
- techpineapple 1y ago
  Additionally, I wonder if any of this fixes the fact that anomaly detection in alerting is traditionally a really hard problem, and one I’ve hardly seen done well. Of any set of packaged or recommended alerts, I probably only use 1% of them because anomalies are often the norm.
  - tptacek 1y ago
    That reinforces his argument.
    - techpineapple 1y ago
      If he stopped at “LLms can help rca’s, I’d agree with you.
  - zer00eyz 1y ago
    > because anomalies are often the norm You fix these issues or you tune your alert system to make it clear that they aren't actionable. Otherwise you end up turning them off so your system doesn't turn into the boy who cried wolf (or worse teams learn to ignore it and it becomes useless) Bayesian filters, and basic dirivative functions (think math) can do a lot to tame output from these systems. These arent "product features" so in most orgs they dont get the attention they need or deserve.
    - zdragnar 1y ago
      > or worse teams learn to ignore it and it becomes useless This is basically every team I've worked with. Product wants new features, and doesn't want to spend on existing features. Hurry up and write new stuff! Ignore problems and they'll go away! Also: I've already reported this bug! Why haven't the developers fixed it is yet?
- tptacek 1y ago
  I'm not sure I understand the question. He's writing from the vantage point of someone with a large oTel deployment; that's the data he has to work with. Honeycomb has an MCP server. Instead of him clicking around Honeycomb and making inferences from the data and deciding where to drill down, an LLM did that, and found the right answer quicker than a human would have. Where's the extra expense here? The $0.60 he spent on LLM calls with his POC agent?
  - danpalmer 1y ago
    I think the implication is that if you have that graph you’re already half way towards a solution because you know there’s a problem. In terms of _identifying the problems_, shoving all your data into an LLM to spot irregularities would be exceptionally expensive vs traditional alerting, even though it may be much more capable at spotting potential issues without explicit alerting thresholds being set up.
    - tptacek 1y ago
      Look, I like Honeycomb a lot and we're dependent on it for parts of our orchestrator. It's great; it accelerates investigations. But even with Honeycomb, we are sitting on an absolute mountain of telemetry data, in logs, in metrics, and in our trace indices at Honeycomb. We can solve problems by searching and drilling down into all those data sources; that's how everybody solves problems. But solving problems takes time. Just having the data in the graph does not mean we're near a solution! An LLM agent can chase down hypotheses, several at a time, and present them with collected data and plausible narratives. It'll miss things and come up with wild goose chases, but so do people during incidents.
      - datadrivenangel 1y ago
        I think the point is that all your telemetry data is going to be very expensive.
        danpalmer 1y ago
- zer00eyz 1y ago
  Also we need to talk about what should be logged and where. There seem to be two schools of thought, just enough to tell something is wrong but not what it is - OR - you get to drink from the firehose. And most orgs go from the first to the second. As to where, well thats at the hardware/vm/container level, and mirror and extend what it does. Nothing worse than 20 different ideas of how to log and rotate and trying to figure out who did what, when where and why. If you can't match a log entry to a running environment... well. I weep quietly inside when some or all of this goes through one, or several S3 buckets for no good reason.
stlava 1y ago
I feel that if you need an LLM to help pivot between existing data it just means the operability tool has gaps in user functionality. This is by far my biggest gripe with DataDog today. All the data is there but going from database query to front end traces should be easy but is not. Sure we can use an LLM but I can for now click around faster (if those breadcrumbs exist) than it can reason. Also the LLM would only point to a direction and I’m still going to have to use the UI to confirm.
- tptacek 1y ago
  One of the interesting things an agent can do that no individual telemetry tool does effectively is make deductions and integrate information across data sources. It's a big open challenge for us here; in any given incident, we're looking at Honeycomb traces, OpenSearch for system logs, and Prometheus metrics in a VictoriaMetrics cluster. Given tool calls for each of these data sources, an agent can generate useful integrated hypotheses without any direct integration between the data sources. That's pretty remarkable.
  - dengolius 1y ago
    VictoriaMetrics team member here. Have you had a chance to play with VictoriaLogs? If not, then I highly recommend to test it. Our team is also working on implementing traces in top of Jaeger and VictoriaLogs, see https://victoriametrics.com/blog/dev-note-distributed-tracing-with-victorialogs/ https://victoriametrics.com/blog/dev-note-distributed-tracin...
- JyB 1y ago
  The whole point of Datadog is that you can seamlessly go between products for the same event source. Doesn’t it just highlight a setup issue?
physix 1y ago
I'd like to see the long list of companies that are in the process of being le cooked.
ok_dad 1y ago
"Get AI to do stuff you can already do with a little work and some experts in the field." What a good business strategy! I could post this comment on 80% of the AI application companies today, sadly.
- tptacek 1y ago
  I think you think this is a dunk, but "some experts in [this] field" are extraordinarily expensive. If you can actually do this, it's no wonder you're seeing so many flimsy AI companies.
  - NewJazz 1y ago
    I think the commenter might have been saying that you need experts I'm the field to leverage AI here, in which case your response is supporting their point.
    - tptacek 1y ago
      "you can already do with" implies otherwise.
      - NewJazz 1y ago
        It is ambiguous and arguably poorly worded... Like I said I'm not really sure how to interpret the comment. (Get AI to do stuff (you can already do) with a little work and some experts in the field). (Get AI to do stuff (you can already do with a little work and some experts in the field)).
        deleted 1y ago
        [deleted]
AdieuToLogic 1y ago
This post is a thinly veiled marketing promo. Here's why. Skip to the summary section titled "Fast feedback is the only feedback" and its first assertion: ... the only thing that really matters is fast, tight feedback loops at every stage of development and operations. This is industry dogma generally considered "best practice" and sets up the subsequent straw man: AI thrives on speed—it'll outrun you every time. False. "AI thrives" on many things, but "speed" is not one of them. Note the false consequence ("it'll outrun you every time") used to set up the the epitome of vacuous sales pitch drivel: To succeed, you need tools that move at the speed of AI as well. I hope there's a way I can possibly "move at the speed of AI"... Honeycomb's entire modus operandi is predicated on fast feedback loops, collaborative knowledge sharing, and treating everything as an experiment. We’re built for the future that’s here today, on a platform that allows us to be the best tool for tomorrow. This is as subtle as a sledgehammer to the forehead. What's even funnier is the lame attempt to appear objective after all of this: I’m also not really in the business of making predictions. Really? Did the author read anything they wrote before this point?
- pgwhalen 1y ago
  Is it even attempting to be veiled at all? You know you’re reading a company’s blog post, written about a feature the company is building for their product, right? It is explicitly marketing.
  - AdieuToLogic 1y ago
    > Is it even attempting to be veiled at all? I do believe the veil is at best "thin." Perhaps I was being too generous given the post starts with: New abstractions and techniques for software development and deployment gain traction, those abstractions make software more accessible by hiding complexity, and that complexity requires new ways to monitor and measure what’s happening. We build tools like dashboards, adaptive alerting, and dynamic sampling. All of these help us compress the sheer amount of stuff happening into something that’s comprehensible to our human intelligence. In AI, I see the death of this paradigm. It’s already real, it’s already here, and it’s going to fundamentally change the way we approach systems design and operation in the future. Maybe I should have detected the utterly condescending phrase, "something that’s comprehensible to our human intelligence."
zug_zug 1y ago
As somebody who's good at RCA, I'm worried all my embarrassed coworkers are going to take at face value a tool that's confidently incorrect 10% of the time and screw stuff up more instead of having to admit they don't know something publicly. It'd be less bad if the tool came to a conclusion, then looked for data to disprove that interpretation, and then made a more reliably argument or admitted its uncertainty.
- jakogut 1y ago
  You can achieve a good amount of this with system prompts. I've actually had good success using LLMs to craft effective system prompts and custom instructions to get more rigorous and well researched answers by default. One I use with ChatGPT currently is: > Prioritize substance, clarity, and depth. Challenge all my proposals, designs, and conclusions as hypotheses to be tested. Sharpen follow-up questions for precision, surfacing hidden assumptions, trade offs, and failure modes early. Default to terse, logically structured, information-dense responses unless detailed exploration is required. Skip unnecessary praise unless grounded in evidence. Explicitly acknowledge uncertainty when applicable. Always propose at least one alternative framing. Accept critical debate as normal and preferred. Treat all factual claims as provisional unless cited or clearly justified. Cite when appropriate. Acknowledge when claims rely on inference or incomplete information. Favor accuracy over sounding certain.
  - j_bum 1y ago
    Do you just add this to your “instructions” section? And what type of questions do you ask the model? Thanks for sharing
    - jakogut 1y ago
      Yes, with ChatGPT, I added that paragraph as custom instructions under personalization. I ask a wide variety of things, from what a given plant is deficient in based on a photo, to wireless throughout optimization (went from 600 Mbps to 944 Mbps in one hour of tuning). I use models for discovering new commands, tools, and workflows, interpreting command output, and learning new keywords for deeper and more rigorous research using more conventional methods. I rubber duck with it, explaining technical problems, my hypotheses, and iterating over experiments until arriving at a solution, creating a log in the process. The model is often wrong, but it's also often right, and used the way I do, it's quickly apparent when it's wrong. I've used ChatGPT's memory feature to extract questions from previous chats that have already been answered to test the quality and usability of local models like Gemma3, as well as craft new prompts in adjacent topics. Prompts that are high leverage, compact, and designed to trip up models that are underpowered or over quantized. For example: >> "Why would toggling a GPIO in a tight loop not produce a square wave on the pin?" > Tests: hardware debounce, GPIO write latency, MMIO vs cache, bus timing. >> "Why is initrd integrity important for disk encryption with TPM sealing?" > Tests: early boot, attack surface, initramfs tampering vectors. >> "Why would a Vulkan compute shader run slower on an iGPU than a CPU?" > Tests: memory bandwidth vs cache locality, driver maturity, PCIe vs UMA. >> "Why is async compute ineffective on some GPUs?" > Tests: queue scheduling, preemption granularity, workload balance. >> "Why might a PID loop overshoot more when sensor update rate decreases?" > Tests: delayed ACK, bufferbloat, congestion control tuning. >> "How can TCP experience high latency even with low packet loss?" > Tests: delayed ACK, bufferbloat, congestion control tuning. >> "How can increasing an internal combustion engine's compression ratio improve both torque and efficiency?" > Tests: thermodynamics, combustion behavior, fuel octane interaction. >> "How can increasing memory in a server slow it down?" > Tests: NUMA balancing, page table size, cache dilution. >> "Why would turning off SMT increase single-threaded performance on some CPUs?" > Tests: resource contention, L1/L2 pressure, scheduling policy.
resonious 1y ago
The title is a bit overly dramatic. You still need all of your existing observability tools, so nothing is ending. You just might not need to spend quite as much time building and staring at graphs. It's the same effect LLMs are having on everything, it seems. They can help you get faster at something you already know how to do (and help you learn how to do something!), but they don't seem to outright replace any particular skill.
- sakesun 1y ago
  1. help you get faster at something you already know how to do 2. and help you learn how to do something! This is the second time I heard this conclusion today. Using inference to do 2. and then getting superpower in doing 1., this is probably the right way to go forward.
  - germandiago 1y ago
    This is exactly how I have been using it. Which takes me to the question: who would you hire: 1. an expert for salary X? 2. a not-so-expert for 0.4x salary + AI tool that can do well enough? I suspect that, unless there are specific requirements, 2. tends to win.
    - Seb-C 1y ago
      2 might win until the damage he does becomes visible enough. Then 1 wins.
  - autoexec 1y ago
    > 2. and help you learn how to do something! I think it's more likely that people will learn far less. AI will regurgitate an answer to their problem and people will use it without understanding or even verifying if it is correct so long as it looks good. All the opportunities a person would have to discover and learn about things outside of the narrow scope they initially set their sights on will be lost. Someone will ask AI to do X and they get their copy/paste solution so there's nothing to learn or think about. They'll never discover things like why X isn't such a good idea or that Y does the job even better. They'll never learn about Z either, the random thing that they'd have stumbled upon while looking for more info about X.
stego-tech 1y ago
Again, sales pitch aside, this is one of the handful of valuable LLM applications out there. Monitoring and observability have long been the exclusive domains of SRE teams in large orgs while simultaneously out of reach to smaller orgs (speaking strictly from an IT perspective, NOT dev), because identifying valuable metrics and carving up heartbeats and baselines for them is something that takes a lot of time, specialized tooling, extensive dev environments to validate changes, and change controls to ensure you don’t torch production. With LLMs trained on the most popular tools out there, this gives IT teams short on funds or expertise the ability to finally implement “big boy” observability and monitoring deployments built on more open frameworks or tools, rather than yet-another-expensive-subscription. For usable dashboards and straightforward observability setups, LLMs are a kind of god-send for IT folks who can troubleshoot and read documentation, but lack the time for a “deep dive” on every product suite the CIO wants to shove down our throats. Add in an ability to at least give a suggested cause when sending a PagerDuty alert, and you’ve got a revolution in observability for SMBs and SMEs.
- JimBlackwood 1y ago
  Agreed! I see huge gains for small SRE teams aswell. I’m in a team of two with hundreds of bare metal machines under management - if issues pop up it can be stressful to quickly narrow your search window to a culprit. I’ve been contemplating writing an MCP to help out with this, the future seems bright in this regard. Plenty of times when issues have been present for a while before creating errors, aswell. LLM’s again can help with this.
- chupasaurus 1y ago
  > identifying valuable metrics and carving up heartbeats and baselines for them The first problem is out of reach for LLMs, the other 2 are trivial with convolutional NNs for a long time. > extensive dev environments to validate changes, and change controls to ensure you don’t torch production Are out of scope for observability.
  - stego-tech 1y ago
    Not for small environments they’re not. If you say “observability” to a big Fortune company, they peg it to a narrow, specifically defined team and goal; that same word to a SMB/E is “oh, you mean monitoring and change control”. I used the word correctly for the environments I was describing. They’re just not accurate for larger, more siloed orgs.
mediumsmart 1y ago
I thought the article was about the end of observability of the real world as we knew it and was puzzled why they felt fine.
deleted 1y ago
[deleted]
kacesensitive 1y ago
LLMs won't replace observability, but they absolutely change the game. Asking "why is latency spiking" and getting a coherent root cause in seconds is powerful. You still need good telemetry, but this shifts the value from visualizing data to explaining it.
- Nathanba 1y ago
  I was initially agreeing with the article but it's a clever marketing piece. Nothing changes, these graphs were already really easy to read and if they weren't then they should be. You should already be capable of zooming into your latency spike within seconds and clicking on it and seeing which method was slow. So asking the AI will be more comfortable but it doesn't change anything.
- ActorNightly 1y ago
  LLMS are basically just like higher level programming tools - knowing how to utilize them is the key. Best practice is not to depend on them for correctness, but instead utilize them as automatic maps from data->action that you would otherwise have to write manually. For example, I wrote my own MCP server in Python that basically makes it easy to record web browser activities and replay them using playwright. When I have to look at logs or inspect metrics, I record the workflow, and import it as an MCP tool. Then I keep a prompt file where I record what the task was, tool name, description of the output, and what is the final answer given an output. So now, instead of doing the steps manually, I basically just ask Claude to do things. At some point, I am going to integrate real time voice recording and trigger on "Hey Claude" so I don't even have to type. The only thing I wish someone would do is basically make a much smaller model with limited training only on things related to computer science, so it can run at high resolution on a single Nvidia card with fast inference.
- jacobsenscott 1y ago
  The problem with LLMs is the answer always sounds right, no matter if it is or isn't. If you already know the answer to a question it is kind of fun to see an LLM get lucky and cobble together a correct answer. But they are otherwise useless - you need to do all the same work you would do anyway to check the LLM's "answer".
RainyDayTmrw 1y ago
I think we are, collectively, greatly underestimating the value of determinism and, conversely, the cost of nondeterminism. I've been trialing a different product with the same sales pitch. It tries to RCE my incidents by correlating graphs. It ends up looking like this page[1], which is a bit hard to explain in words, but both obvious and hilarious when you see it for yourself. [1]: https://tylervigen.com/spurious-correlations https://tylervigen.com/spurious-correlations
- graemep 1y ago
  Its fun, but the point should be well known (i know its not). Time series are very prone to spurious correlations - r² is not useful. Its even worse if you just eyeball a graph. If something changes over time, you need to use appropriate measures.
  - feoren 1y ago
    > r² is not useful People want so badly to have an "objective" measure of truth that they can just pump data into and get a numeric result that "doesn't lie". r², p < 0.05, χ2, etc. It's too far to say these numbers aren't useful at all -- they are -- but we're just never going to be able to avoid the difficult and subjective task of interpreting experimental results in their broader context and reconciling them with our pre-existing knowledge. I think this is why people are so opposed to anything Bayesian: we don't want to have to do that work, to have those arguments with each other about what we believed before the experiment and how strongly. But the more we try to be objective, the more vulnerable we are to false positives and spurious correlations.
  - SOLAR_FIELDS 1y ago
    I agree - the problem here is probably less that the AI makes the mistake and more that it’s just really easy to make this mistake whether you’re AI or human. It’s probably true that the AI selects more correlations that would be obviously irrelevant to the end user though. I, like others doing SRE work, spend a fair amount of time trawling through these types of graphs and it’s very common to see something that looks like a correlation, look closer, then dismiss it due to noise. Time series are also very much subject to MTUP[1]. Something that looks like a correlation or problem at a certain zoom level becomes totally normal behavior when you zoom out, for instance 1: https://en.m.wikipedia.org/wiki/Modifiable_temporal_unit_problem https://en.m.wikipedia.org/wiki/Modifiable_temporal_unit_pro...
favflam 1y ago
I find people relying way too much on AI tools. If I pay someone a salary, they need to understand the actually answer the give me. And their butt needs to be on the line if the answer is wrong. That is the purpose of them getting a salary. It is not just to do the work, but it is to be responsible for the results. AI breaks this in a lot of the use cases I see crop up on ycombinator. If some AI tools outstrips the ability for a human to be in the decision loop, then that AI tool's usefulness is not so great.
- est 1y ago
  AI tools are just like interns. They cost less and do some jobs well, and sometimes they fail.
- tptacek 1y ago
  they need to understand the actually answer the give me OK. And their butt needs to be on the line if the answer is wrong OK. I'm not seeing the problem.
  - roenxi 1y ago
    The problem seems to be a significant number of software programmers not being confident about what they will be doing to justify drawing a salary.
    - wiseowise 1y ago
      I don’t understand the premise: you were hired to do the job, you do the job, tooling improves so you do more job with less resources. It’s a win-win for everyone.
      - layer8 1y ago
        Lower-skilled people will be able to “do the job” with the new tooling (on a level that management believes to be good enough), and doing more job with less resources also means with less human resources. There is no win-win, similar to how there is no win-win for artists who in principle can now produce more “art” with less resources.
satisfice 1y ago
So many engineers feel fine about a tool that they cannot rely upon. Without reliability, nothing else matters, and this AI that can try hypotheses so much faster than me is not reliable. The point is moot.
geraneum 1y ago
> This isn’t a contrived example. I basically asked the agent the same question we’d ask you in a demo, and the agent figured it out with no additional prompts, training, or guidance. It effectively zero-shot a real-world scenario. As I understand, this is a demo they already use and the solution is available. Maybe it should’ve been a contrived example so that we can tell if the solution was not in training data verbatim. Not that it’s not useful what the LLM did but if you announce the death of observability as we know it, you need to show that the tool can generalize.
akrauss 1y ago
I would be interested in reading what tools are made available to the LLM, and how everything is wired together to form an effective analysis loop. It seems like this is a key ingredient here.
- trashtester 1y ago
  For now, the people able to glue all the necessary ingredients together are the same ones who can understand the output if they drill into it. Indeed, these may be the last ones to be fired, as they can become efficient enough to do the jobs of everyone else one day.
yellow_lead 1y ago
Did AI write this entire article? > In AI, I see the death of this paradigm. It’s already real, it’s already here, and it’s going to fundamentally change the way we approach systems design and operation in the future. How is AI analyzing some data the "end of observability as we know it"?
schwede 1y ago
Maybe I’m just a skeptic, but it seems like a software engineer or SRE familiar with the application should be able to come to the conclusion of load testing fairly easily. For sure not as fast like 80 seconds though which is impressive. As noted you still need an engineer to review the data and complete those proposed action items.
devmor 1y ago
As the AI growth cycle stagnates while valuations continue to fly wildly out of control and more and more of the industry switches from hopeful to a bearish sentiment, I’ve started to find this genre of article extremely funny, if not pitiable. Who are you trying to convince with this? It’s not going to work on investors much longer, it’s mostly stopped working on the generically tech-inclined, and it’s never really worked on anyone who understands AI. So who’s left to be suckered by this flowery, desperate prose? Are you just trying to convince yourselves?
vanschelven 1y ago
> New abstractions and techniques... hide complexity, and that complexity requires new ways to monitor and measure. If the abstractions hide complexity so well you need an LLM to untangle them later, maybe you were already on the wrong track. Hiding isn't abstracting, and if your system becomes observable only with AI help, maybe it's not well designed, just well obfuscated. I've written about this before here: https://www.bugsink.com/blog/you-dont-need-application-performance-monitoring/ https://www.bugsink.com/blog/you-dont-need-application-perfo...
neuroelectron 1y ago
This would have been really nice to have when I was in Ops. Running MapReduce on logs and looking at dozens of graphs made up most of my working hours. We did eventually get the infrastructure for live filtering but that was just before the entire sector was outsourced.
catlifeonmars 1y ago
Was anyone else just curious about those odd spikes and was disappointed the article didn’t do a deeper dive to explain that unusual shape?
- anentropic 1y ago
  Did you read to the end? It identified them as result of load testing - they were isolated to a single user, against a single endpoint (checkout), user-agent was python/requests, shopping cart qty was unusually large, and they had the 'ramp up then stop' shape of a load test
  - catlifeonmars 1y ago
    I did, but I apparently missed the ramping part. Thanks for the summary.
heinrichhartman 1y ago
> New Relic did this for the Rails revolution, Datadog did it for the rise of AWS, and Honeycomb led the way for OpenTelemetry. I find this reading of history of OTel highly biased. OpenTelemetry was born as the Merge of OpenCensus (initiated by Google) and OpenTracing (initiated by LightStep): https://opensource.googleblog.com/2019/05/opentelemetry-merger-of-opencensus-and.html?utm_source=chatgpt.com https://opensource.googleblog.com/2019/05/opentelemetry-merg... > The seed governance committee is composed of representatives from Google, Lightstep, Microsoft, and Uber, and more organizations are getting involved every day. Honeycomb has for sure had valuable code & community contributions and championed the technology adoption, but they are very far from "leading the way".
- loevborg 1y ago
  As someone who recently adopted Honeycomb, it really is an amazing tool. Especially with otel auto-instrumentation, you get insights within a few hours. The dashboard / query features are very powerful and obviously stem from a deep philosophical understanding of observability. My team was shocked at how good the tools is. Datadog by contrast seems to be driven by marketing and companies having a "observability" checkbox to tick.
  - ayewo 1y ago
    Which programming languages are you using with the OTel auto-instrumentation feature?
    - loevborg 1y ago
      Node and Python. It's amazing how much works out of the box - express routes, http queries, dns queries, the list goes on
Kiyo-Lynn 1y ago
I used to think that monitoring and alerting systems were just there to help you quickly and directly see the problems.But as the systems grew more complex, I found that the dashboards and alerts became overwhelming, and I often couldn’t figure out the root cause of the issue. Recently, I started using AI to help with analysis, and I found that it can give me clues in a few seconds that I might have spent half a day searching for. While it's much more efficient, sometimes I worry that, even though AI makes problem-solving easier, we might be relying too much on these tools and losing our own ability to judge and analyze.
- vjerancrnjak 1y ago
  Yes, at any particular task you will be better than AI. We somehow forget that none of these systems are better than expert humans. If you rely on a tool, you might never develop the skills. Some skills are more worth than others. You won’t even have the experience to know which ones as well.
- globular-toast 1y ago
  I would argue that people have been relying on observability tools too much rather than designing systems that are understandable in the first place.
- germandiago 1y ago
  This is 100% true and also my experience. However many companies will just care about how fast you deliver a solutiom, not about how much you are learning. They do not care anymore. The speed of the productive process is critical to them in many jobs.
dgellow 1y ago
Sort of related: using Claude code with the gcloud CLI, only allowing read only commands (and of course no ssh), and with supervision, is such a superpower. I don’t think I can go back to debugging my infra manually. It’s like all use of Claude code, not a fire and forget, you have to guide and correct it, but that’s so much faster and easier than dealing directly with the mess GCP APIs is
- benterix 1y ago
  > only allowing read only commands Out of curiosity, how do you do that? I have no experience with this tool, not I would ever thought to use it for infra, but you made me curious.
  - plucas 1y ago
    My assumption is by creating a service account with limited privileges and activating it for gcloud when running this.
    - benterix 1y ago
      Yes, in the meantime I figured out it's basically assigning it the Reader IAM role in GCP.
captainbland 1y ago
Gripe: enshittification is tangential to performance concerns. It doesn't just mean software getting bad, it means software which may be technically accomplished doing things which are bad for users in service of creating ROI for investors after a period of market share building.
gilbetron 1y ago
There's a bit of a flaw in the "don't need graphs and UIs to look at your data" premise behind this article: sure, LLMs will be great ... when the work great. When they fail, you need a human there to figure it out and they will still need the graphs. Furthermore, while graphing and visualization are definitely tough, complex parts about observability, gathering the data and storing it in forms to meet the complex query demands are really difficult as well. Observability will "go away" once AI is capable of nearly flawlessly determining everything out itself, and then AI will be capable of nearly anything, so the "end of observability" is the end of our culture as we know it (probably not extinction, but more like culture will shift profoundly, and probably painfully). AI will definitely change observability, and that's cool. It already is, but has a long way to go.
nilkn 1y ago
It's not the end of observability as we know it. However, the article also isn't totally off-base. We're almost certain to see a new agentic layer emerge and become increasingly capable for various aspects of SRE, including observability tasks like RCA. However, for this to function, most or even all of the existing observability stack will still be needed. And as long as the hallucination / reliability / trust issues with LLMs remain, human deep dives will remain part of the overall SRE work structure.
pmbauer 1y ago
It must burn a little blogging about an LLM-driven latency analysis _internal demo_ only to have Datadog launch a product in the same space a day later. https://www.datadoghq.com/blog/bits-ai-sre/ https://www.datadoghq.com/blog/bits-ai-sre/
- Rastonbury 1y ago
  And it wasn't just that single AI product either, they announced a bunch including an MCP server and cursor extension https://www.datadoghq.com/blog/dash-2025-new-feature-roundup-keynote/ https://www.datadoghq.com/blog/dash-2025-new-feature-roundup...
nektro 1y ago
lol
robertcary25 1y ago
[dead]

301 points by gpi 1y ago

I feel like the alternate title of this could be “how to 10x your observability costs with this one easy trick”. It didn’t really show a way to get rid of all the graphs, the prompt was “show me why my latency spikes every four hours”. That’s really cool, but in order to generate that prompt you need alerts and graphs. How do you know you’re latency is spiking to generate the prompt? The devil seems to be in the details, but you’re running a whole bunch more compute for anomaly detection and “ Sub-second query performance, unified data storage”, which again sounds like throwing enormous amounts of more money at the problem. I can totally see why this is great for honeycomb though, they’re going to make bank.

techpineapple 1y ago
Additionally, I wonder if any of this fixes the fact that anomaly detection in alerting is traditionally a really hard problem, and one I’ve hardly seen done well. Of any set of packaged or recommended alerts, I probably only use 1% of them because anomalies are often the norm.
- tptacek 1y ago
  That reinforces his argument.
  - techpineapple 1y ago
    If he stopped at “LLms can help rca’s, I’d agree with you.
- zer00eyz 1y ago
  > because anomalies are often the norm You fix these issues or you tune your alert system to make it clear that they aren't actionable. Otherwise you end up turning them off so your system doesn't turn into the boy who cried wolf (or worse teams learn to ignore it and it becomes useless) Bayesian filters, and basic dirivative functions (think math) can do a lot to tame output from these systems. These arent "product features" so in most orgs they dont get the attention they need or deserve.
  - zdragnar 1y ago
    > or worse teams learn to ignore it and it becomes useless This is basically every team I've worked with. Product wants new features, and doesn't want to spend on existing features. Hurry up and write new stuff! Ignore problems and they'll go away! Also: I've already reported this bug! Why haven't the developers fixed it is yet?
tptacek 1y ago
I'm not sure I understand the question. He's writing from the vantage point of someone with a large oTel deployment; that's the data he has to work with. Honeycomb has an MCP server. Instead of him clicking around Honeycomb and making inferences from the data and deciding where to drill down, an LLM did that, and found the right answer quicker than a human would have. Where's the extra expense here? The $0.60 he spent on LLM calls with his POC agent?
- danpalmer 1y ago
  I think the implication is that if you have that graph you’re already half way towards a solution because you know there’s a problem. In terms of _identifying the problems_, shoving all your data into an LLM to spot irregularities would be exceptionally expensive vs traditional alerting, even though it may be much more capable at spotting potential issues without explicit alerting thresholds being set up.
  - tptacek 1y ago
    Look, I like Honeycomb a lot and we're dependent on it for parts of our orchestrator. It's great; it accelerates investigations. But even with Honeycomb, we are sitting on an absolute mountain of telemetry data, in logs, in metrics, and in our trace indices at Honeycomb. We can solve problems by searching and drilling down into all those data sources; that's how everybody solves problems. But solving problems takes time. Just having the data in the graph does not mean we're near a solution! An LLM agent can chase down hypotheses, several at a time, and present them with collected data and plausible narratives. It'll miss things and come up with wild goose chases, but so do people during incidents.
    - datadrivenangel 1y ago
      I think the point is that all your telemetry data is going to be very expensive.
      - danpalmer 1y ago
zer00eyz 1y ago
Also we need to talk about what should be logged and where. There seem to be two schools of thought, just enough to tell something is wrong but not what it is - OR - you get to drink from the firehose. And most orgs go from the first to the second. As to where, well thats at the hardware/vm/container level, and mirror and extend what it does. Nothing worse than 20 different ideas of how to log and rotate and trying to figure out who did what, when where and why. If you can't match a log entry to a running environment... well. I weep quietly inside when some or all of this goes through one, or several S3 buckets for no good reason.

stlava 1y ago

I feel that if you need an LLM to help pivot between existing data it just means the operability tool has gaps in user functionality. This is by far my biggest gripe with DataDog today. All the data is there but going from database query to front end traces should be easy but is not. Sure we can use an LLM but I can for now click around faster (if those breadcrumbs exist) than it can reason. Also the LLM would only point to a direction and I’m still going to have to use the UI to confirm.

tptacek 1y ago
One of the interesting things an agent can do that no individual telemetry tool does effectively is make deductions and integrate information across data sources. It's a big open challenge for us here; in any given incident, we're looking at Honeycomb traces, OpenSearch for system logs, and Prometheus metrics in a VictoriaMetrics cluster. Given tool calls for each of these data sources, an agent can generate useful integrated hypotheses without any direct integration between the data sources. That's pretty remarkable.
- dengolius 1y ago
  VictoriaMetrics team member here. Have you had a chance to play with VictoriaLogs? If not, then I highly recommend to test it. Our team is also working on implementing traces in top of Jaeger and VictoriaLogs, see https://victoriametrics.com/blog/dev-note-distributed-tracing-with-victorialogs/ https://victoriametrics.com/blog/dev-note-distributed-tracin...
JyB 1y ago
The whole point of Datadog is that you can seamlessly go between products for the same event source. Doesn’t it just highlight a setup issue?

physix 1y ago

I'd like to see the long list of companies that are in the process of being le cooked.

ok_dad 1y ago

"Get AI to do stuff you can already do with a little work and some experts in the field." What a good business strategy! I could post this comment on 80% of the AI application companies today, sadly.

tptacek 1y ago
I think you think this is a dunk, but "some experts in [this] field" are extraordinarily expensive. If you can actually do this, it's no wonder you're seeing so many flimsy AI companies.
- NewJazz 1y ago
  I think the commenter might have been saying that you need experts I'm the field to leverage AI here, in which case your response is supporting their point.
  - tptacek 1y ago
    "you can already do with" implies otherwise.
    - NewJazz 1y ago
      It is ambiguous and arguably poorly worded... Like I said I'm not really sure how to interpret the comment. (Get AI to do stuff (you can already do) with a little work and some experts in the field). (Get AI to do stuff (you can already do with a little work and some experts in the field)).
      - deleted 1y ago
        [deleted]

AdieuToLogic 1y ago

This post is a thinly veiled marketing promo. Here's why. Skip to the summary section titled "Fast feedback is the only feedback" and its first assertion: ... the only thing that really matters is fast, tight feedback loops at every stage of development and operations. This is industry dogma generally considered "best practice" and sets up the subsequent straw man: AI thrives on speed—it'll outrun you every time. False. "AI thrives" on many things, but "speed" is not one of them. Note the false consequence ("it'll outrun you every time") used to set up the the epitome of vacuous sales pitch drivel: To succeed, you need tools that move at the speed of AI as well. I hope there's a way I can possibly "move at the speed of AI"... Honeycomb's entire modus operandi is predicated on fast feedback loops, collaborative knowledge sharing, and treating everything as an experiment. We’re built for the future that’s here today, on a platform that allows us to be the best tool for tomorrow. This is as subtle as a sledgehammer to the forehead. What's even funnier is the lame attempt to appear objective after all of this: I’m also not really in the business of making predictions. Really? Did the author read anything they wrote before this point?

pgwhalen 1y ago
Is it even attempting to be veiled at all? You know you’re reading a company’s blog post, written about a feature the company is building for their product, right? It is explicitly marketing.
- AdieuToLogic 1y ago
  > Is it even attempting to be veiled at all? I do believe the veil is at best "thin." Perhaps I was being too generous given the post starts with: New abstractions and techniques for software development and deployment gain traction, those abstractions make software more accessible by hiding complexity, and that complexity requires new ways to monitor and measure what’s happening. We build tools like dashboards, adaptive alerting, and dynamic sampling. All of these help us compress the sheer amount of stuff happening into something that’s comprehensible to our human intelligence. In AI, I see the death of this paradigm. It’s already real, it’s already here, and it’s going to fundamentally change the way we approach systems design and operation in the future. Maybe I should have detected the utterly condescending phrase, "something that’s comprehensible to our human intelligence."

zug_zug 1y ago

As somebody who's good at RCA, I'm worried all my embarrassed coworkers are going to take at face value a tool that's confidently incorrect 10% of the time and screw stuff up more instead of having to admit they don't know something publicly. It'd be less bad if the tool came to a conclusion, then looked for data to disprove that interpretation, and then made a more reliably argument or admitted its uncertainty.

jakogut 1y ago
You can achieve a good amount of this with system prompts. I've actually had good success using LLMs to craft effective system prompts and custom instructions to get more rigorous and well researched answers by default. One I use with ChatGPT currently is: > Prioritize substance, clarity, and depth. Challenge all my proposals, designs, and conclusions as hypotheses to be tested. Sharpen follow-up questions for precision, surfacing hidden assumptions, trade offs, and failure modes early. Default to terse, logically structured, information-dense responses unless detailed exploration is required. Skip unnecessary praise unless grounded in evidence. Explicitly acknowledge uncertainty when applicable. Always propose at least one alternative framing. Accept critical debate as normal and preferred. Treat all factual claims as provisional unless cited or clearly justified. Cite when appropriate. Acknowledge when claims rely on inference or incomplete information. Favor accuracy over sounding certain.
- j_bum 1y ago
  Do you just add this to your “instructions” section? And what type of questions do you ask the model? Thanks for sharing
  - jakogut 1y ago
    Yes, with ChatGPT, I added that paragraph as custom instructions under personalization. I ask a wide variety of things, from what a given plant is deficient in based on a photo, to wireless throughout optimization (went from 600 Mbps to 944 Mbps in one hour of tuning). I use models for discovering new commands, tools, and workflows, interpreting command output, and learning new keywords for deeper and more rigorous research using more conventional methods. I rubber duck with it, explaining technical problems, my hypotheses, and iterating over experiments until arriving at a solution, creating a log in the process. The model is often wrong, but it's also often right, and used the way I do, it's quickly apparent when it's wrong. I've used ChatGPT's memory feature to extract questions from previous chats that have already been answered to test the quality and usability of local models like Gemma3, as well as craft new prompts in adjacent topics. Prompts that are high leverage, compact, and designed to trip up models that are underpowered or over quantized. For example: >> "Why would toggling a GPIO in a tight loop not produce a square wave on the pin?" > Tests: hardware debounce, GPIO write latency, MMIO vs cache, bus timing. >> "Why is initrd integrity important for disk encryption with TPM sealing?" > Tests: early boot, attack surface, initramfs tampering vectors. >> "Why would a Vulkan compute shader run slower on an iGPU than a CPU?" > Tests: memory bandwidth vs cache locality, driver maturity, PCIe vs UMA. >> "Why is async compute ineffective on some GPUs?" > Tests: queue scheduling, preemption granularity, workload balance. >> "Why might a PID loop overshoot more when sensor update rate decreases?" > Tests: delayed ACK, bufferbloat, congestion control tuning. >> "How can TCP experience high latency even with low packet loss?" > Tests: delayed ACK, bufferbloat, congestion control tuning. >> "How can increasing an internal combustion engine's compression ratio improve both torque and efficiency?" > Tests: thermodynamics, combustion behavior, fuel octane interaction. >> "How can increasing memory in a server slow it down?" > Tests: NUMA balancing, page table size, cache dilution. >> "Why would turning off SMT increase single-threaded performance on some CPUs?" > Tests: resource contention, L1/L2 pressure, scheduling policy.

resonious 1y ago

The title is a bit overly dramatic. You still need all of your existing observability tools, so nothing is ending. You just might not need to spend quite as much time building and staring at graphs. It's the same effect LLMs are having on everything, it seems. They can help you get faster at something you already know how to do (and help you learn how to do something!), but they don't seem to outright replace any particular skill.

sakesun 1y ago
1. help you get faster at something you already know how to do 2. and help you learn how to do something! This is the second time I heard this conclusion today. Using inference to do 2. and then getting superpower in doing 1., this is probably the right way to go forward.
- germandiago 1y ago
  This is exactly how I have been using it. Which takes me to the question: who would you hire: 1. an expert for salary X? 2. a not-so-expert for 0.4x salary + AI tool that can do well enough? I suspect that, unless there are specific requirements, 2. tends to win.
  - Seb-C 1y ago
    2 might win until the damage he does becomes visible enough. Then 1 wins.
- autoexec 1y ago
  > 2. and help you learn how to do something! I think it's more likely that people will learn far less. AI will regurgitate an answer to their problem and people will use it without understanding or even verifying if it is correct so long as it looks good. All the opportunities a person would have to discover and learn about things outside of the narrow scope they initially set their sights on will be lost. Someone will ask AI to do X and they get their copy/paste solution so there's nothing to learn or think about. They'll never discover things like why X isn't such a good idea or that Y does the job even better. They'll never learn about Z either, the random thing that they'd have stumbled upon while looking for more info about X.

stego-tech 1y ago

Again, sales pitch aside, this is one of the handful of valuable LLM applications out there. Monitoring and observability have long been the exclusive domains of SRE teams in large orgs while simultaneously out of reach to smaller orgs (speaking strictly from an IT perspective, NOT dev), because identifying valuable metrics and carving up heartbeats and baselines for them is something that takes a lot of time, specialized tooling, extensive dev environments to validate changes, and change controls to ensure you don’t torch production. With LLMs trained on the most popular tools out there, this gives IT teams short on funds or expertise the ability to finally implement “big boy” observability and monitoring deployments built on more open frameworks or tools, rather than yet-another-expensive-subscription. For usable dashboards and straightforward observability setups, LLMs are a kind of god-send for IT folks who can troubleshoot and read documentation, but lack the time for a “deep dive” on every product suite the CIO wants to shove down our throats. Add in an ability to at least give a suggested cause when sending a PagerDuty alert, and you’ve got a revolution in observability for SMBs and SMEs.

JimBlackwood 1y ago
Agreed! I see huge gains for small SRE teams aswell. I’m in a team of two with hundreds of bare metal machines under management - if issues pop up it can be stressful to quickly narrow your search window to a culprit. I’ve been contemplating writing an MCP to help out with this, the future seems bright in this regard. Plenty of times when issues have been present for a while before creating errors, aswell. LLM’s again can help with this.
chupasaurus 1y ago
> identifying valuable metrics and carving up heartbeats and baselines for them The first problem is out of reach for LLMs, the other 2 are trivial with convolutional NNs for a long time. > extensive dev environments to validate changes, and change controls to ensure you don’t torch production Are out of scope for observability.
- stego-tech 1y ago
  Not for small environments they’re not. If you say “observability” to a big Fortune company, they peg it to a narrow, specifically defined team and goal; that same word to a SMB/E is “oh, you mean monitoring and change control”. I used the word correctly for the environments I was describing. They’re just not accurate for larger, more siloed orgs.

mediumsmart 1y ago

I thought the article was about the end of observability of the real world as we knew it and was puzzled why they felt fine.

deleted 1y ago

[deleted]

kacesensitive 1y ago

LLMs won't replace observability, but they absolutely change the game. Asking "why is latency spiking" and getting a coherent root cause in seconds is powerful. You still need good telemetry, but this shifts the value from visualizing data to explaining it.

Nathanba 1y ago
I was initially agreeing with the article but it's a clever marketing piece. Nothing changes, these graphs were already really easy to read and if they weren't then they should be. You should already be capable of zooming into your latency spike within seconds and clicking on it and seeing which method was slow. So asking the AI will be more comfortable but it doesn't change anything.
ActorNightly 1y ago
LLMS are basically just like higher level programming tools - knowing how to utilize them is the key. Best practice is not to depend on them for correctness, but instead utilize them as automatic maps from data->action that you would otherwise have to write manually. For example, I wrote my own MCP server in Python that basically makes it easy to record web browser activities and replay them using playwright. When I have to look at logs or inspect metrics, I record the workflow, and import it as an MCP tool. Then I keep a prompt file where I record what the task was, tool name, description of the output, and what is the final answer given an output. So now, instead of doing the steps manually, I basically just ask Claude to do things. At some point, I am going to integrate real time voice recording and trigger on "Hey Claude" so I don't even have to type. The only thing I wish someone would do is basically make a much smaller model with limited training only on things related to computer science, so it can run at high resolution on a single Nvidia card with fast inference.
jacobsenscott 1y ago
The problem with LLMs is the answer always sounds right, no matter if it is or isn't. If you already know the answer to a question it is kind of fun to see an LLM get lucky and cobble together a correct answer. But they are otherwise useless - you need to do all the same work you would do anyway to check the LLM's "answer".

RainyDayTmrw 1y ago

I think we are, collectively, greatly underestimating the value of determinism and, conversely, the cost of nondeterminism. I've been trialing a different product with the same sales pitch. It tries to RCE my incidents by correlating graphs. It ends up looking like this page[1], which is a bit hard to explain in words, but both obvious and hilarious when you see it for yourself. [1]: https://tylervigen.com/spurious-correlations https://tylervigen.com/spurious-correlations

graemep 1y ago
Its fun, but the point should be well known (i know its not). Time series are very prone to spurious correlations - r² is not useful. Its even worse if you just eyeball a graph. If something changes over time, you need to use appropriate measures.
- feoren 1y ago
  > r² is not useful People want so badly to have an "objective" measure of truth that they can just pump data into and get a numeric result that "doesn't lie". r², p < 0.05, χ2, etc. It's too far to say these numbers aren't useful at all -- they are -- but we're just never going to be able to avoid the difficult and subjective task of interpreting experimental results in their broader context and reconciling them with our pre-existing knowledge. I think this is why people are so opposed to anything Bayesian: we don't want to have to do that work, to have those arguments with each other about what we believed before the experiment and how strongly. But the more we try to be objective, the more vulnerable we are to false positives and spurious correlations.
- SOLAR_FIELDS 1y ago
  I agree - the problem here is probably less that the AI makes the mistake and more that it’s just really easy to make this mistake whether you’re AI or human. It’s probably true that the AI selects more correlations that would be obviously irrelevant to the end user though. I, like others doing SRE work, spend a fair amount of time trawling through these types of graphs and it’s very common to see something that looks like a correlation, look closer, then dismiss it due to noise. Time series are also very much subject to MTUP[1]. Something that looks like a correlation or problem at a certain zoom level becomes totally normal behavior when you zoom out, for instance 1: https://en.m.wikipedia.org/wiki/Modifiable_temporal_unit_problem https://en.m.wikipedia.org/wiki/Modifiable_temporal_unit_pro...

favflam 1y ago

I find people relying way too much on AI tools. If I pay someone a salary, they need to understand the actually answer the give me. And their butt needs to be on the line if the answer is wrong. That is the purpose of them getting a salary. It is not just to do the work, but it is to be responsible for the results. AI breaks this in a lot of the use cases I see crop up on ycombinator. If some AI tools outstrips the ability for a human to be in the decision loop, then that AI tool's usefulness is not so great.

est 1y ago
AI tools are just like interns. They cost less and do some jobs well, and sometimes they fail.
tptacek 1y ago
they need to understand the actually answer the give me OK. And their butt needs to be on the line if the answer is wrong OK. I'm not seeing the problem.
- roenxi 1y ago
  The problem seems to be a significant number of software programmers not being confident about what they will be doing to justify drawing a salary.
  - wiseowise 1y ago
    I don’t understand the premise: you were hired to do the job, you do the job, tooling improves so you do more job with less resources. It’s a win-win for everyone.
    - layer8 1y ago
      Lower-skilled people will be able to “do the job” with the new tooling (on a level that management believes to be good enough), and doing more job with less resources also means with less human resources. There is no win-win, similar to how there is no win-win for artists who in principle can now produce more “art” with less resources.

satisfice 1y ago

So many engineers feel fine about a tool that they cannot rely upon. Without reliability, nothing else matters, and this AI that can try hypotheses so much faster than me is not reliable. The point is moot.

geraneum 1y ago

> This isn’t a contrived example. I basically asked the agent the same question we’d ask you in a demo, and the agent figured it out with no additional prompts, training, or guidance. It effectively zero-shot a real-world scenario. As I understand, this is a demo they already use and the solution is available. Maybe it should’ve been a contrived example so that we can tell if the solution was not in training data verbatim. Not that it’s not useful what the LLM did but if you announce the death of observability as we know it, you need to show that the tool can generalize.

akrauss 1y ago

I would be interested in reading what tools are made available to the LLM, and how everything is wired together to form an effective analysis loop. It seems like this is a key ingredient here.

trashtester 1y ago
For now, the people able to glue all the necessary ingredients together are the same ones who can understand the output if they drill into it. Indeed, these may be the last ones to be fired, as they can become efficient enough to do the jobs of everyone else one day.

yellow_lead 1y ago

Did AI write this entire article? > In AI, I see the death of this paradigm. It’s already real, it’s already here, and it’s going to fundamentally change the way we approach systems design and operation in the future. How is AI analyzing some data the "end of observability as we know it"?

schwede 1y ago

Maybe I’m just a skeptic, but it seems like a software engineer or SRE familiar with the application should be able to come to the conclusion of load testing fairly easily. For sure not as fast like 80 seconds though which is impressive. As noted you still need an engineer to review the data and complete those proposed action items.

devmor 1y ago

As the AI growth cycle stagnates while valuations continue to fly wildly out of control and more and more of the industry switches from hopeful to a bearish sentiment, I’ve started to find this genre of article extremely funny, if not pitiable. Who are you trying to convince with this? It’s not going to work on investors much longer, it’s mostly stopped working on the generically tech-inclined, and it’s never really worked on anyone who understands AI. So who’s left to be suckered by this flowery, desperate prose? Are you just trying to convince yourselves?

vanschelven 1y ago

> New abstractions and techniques... hide complexity, and that complexity requires new ways to monitor and measure. If the abstractions hide complexity so well you need an LLM to untangle them later, maybe you were already on the wrong track. Hiding isn't abstracting, and if your system becomes observable only with AI help, maybe it's not well designed, just well obfuscated. I've written about this before here: https://www.bugsink.com/blog/you-dont-need-application-performance-monitoring/ https://www.bugsink.com/blog/you-dont-need-application-perfo...

neuroelectron 1y ago

This would have been really nice to have when I was in Ops. Running MapReduce on logs and looking at dozens of graphs made up most of my working hours. We did eventually get the infrastructure for live filtering but that was just before the entire sector was outsourced.

catlifeonmars 1y ago

Was anyone else just curious about those odd spikes and was disappointed the article didn’t do a deeper dive to explain that unusual shape?

anentropic 1y ago
Did you read to the end? It identified them as result of load testing - they were isolated to a single user, against a single endpoint (checkout), user-agent was python/requests, shopping cart qty was unusually large, and they had the 'ramp up then stop' shape of a load test
- catlifeonmars 1y ago
  I did, but I apparently missed the ramping part. Thanks for the summary.

heinrichhartman 1y ago

> New Relic did this for the Rails revolution, Datadog did it for the rise of AWS, and Honeycomb led the way for OpenTelemetry. I find this reading of history of OTel highly biased. OpenTelemetry was born as the Merge of OpenCensus (initiated by Google) and OpenTracing (initiated by LightStep): https://opensource.googleblog.com/2019/05/opentelemetry-merger-of-opencensus-and.html?utm_source=chatgpt.com https://opensource.googleblog.com/2019/05/opentelemetry-merg... > The seed governance committee is composed of representatives from Google, Lightstep, Microsoft, and Uber, and more organizations are getting involved every day. Honeycomb has for sure had valuable code & community contributions and championed the technology adoption, but they are very far from "leading the way".

loevborg 1y ago
As someone who recently adopted Honeycomb, it really is an amazing tool. Especially with otel auto-instrumentation, you get insights within a few hours. The dashboard / query features are very powerful and obviously stem from a deep philosophical understanding of observability. My team was shocked at how good the tools is. Datadog by contrast seems to be driven by marketing and companies having a "observability" checkbox to tick.
- ayewo 1y ago
  Which programming languages are you using with the OTel auto-instrumentation feature?
  - loevborg 1y ago
    Node and Python. It's amazing how much works out of the box - express routes, http queries, dns queries, the list goes on

Kiyo-Lynn 1y ago

I used to think that monitoring and alerting systems were just there to help you quickly and directly see the problems.But as the systems grew more complex, I found that the dashboards and alerts became overwhelming, and I often couldn’t figure out the root cause of the issue. Recently, I started using AI to help with analysis, and I found that it can give me clues in a few seconds that I might have spent half a day searching for. While it's much more efficient, sometimes I worry that, even though AI makes problem-solving easier, we might be relying too much on these tools and losing our own ability to judge and analyze.

vjerancrnjak 1y ago
Yes, at any particular task you will be better than AI. We somehow forget that none of these systems are better than expert humans. If you rely on a tool, you might never develop the skills. Some skills are more worth than others. You won’t even have the experience to know which ones as well.
globular-toast 1y ago
I would argue that people have been relying on observability tools too much rather than designing systems that are understandable in the first place.
germandiago 1y ago
This is 100% true and also my experience. However many companies will just care about how fast you deliver a solutiom, not about how much you are learning. They do not care anymore. The speed of the productive process is critical to them in many jobs.

dgellow 1y ago

Sort of related: using Claude code with the gcloud CLI, only allowing read only commands (and of course no ssh), and with supervision, is such a superpower. I don’t think I can go back to debugging my infra manually. It’s like all use of Claude code, not a fire and forget, you have to guide and correct it, but that’s so much faster and easier than dealing directly with the mess GCP APIs is

benterix 1y ago
> only allowing read only commands Out of curiosity, how do you do that? I have no experience with this tool, not I would ever thought to use it for infra, but you made me curious.
- plucas 1y ago
  My assumption is by creating a service account with limited privileges and activating it for gcloud when running this.
  - benterix 1y ago
    Yes, in the meantime I figured out it's basically assigning it the Reader IAM role in GCP.

captainbland 1y ago

Gripe: enshittification is tangential to performance concerns. It doesn't just mean software getting bad, it means software which may be technically accomplished doing things which are bad for users in service of creating ROI for investors after a period of market share building.

gilbetron 1y ago

There's a bit of a flaw in the "don't need graphs and UIs to look at your data" premise behind this article: sure, LLMs will be great ... when the work great. When they fail, you need a human there to figure it out and they will still need the graphs. Furthermore, while graphing and visualization are definitely tough, complex parts about observability, gathering the data and storing it in forms to meet the complex query demands are really difficult as well. Observability will "go away" once AI is capable of nearly flawlessly determining everything out itself, and then AI will be capable of nearly anything, so the "end of observability" is the end of our culture as we know it (probably not extinction, but more like culture will shift profoundly, and probably painfully). AI will definitely change observability, and that's cool. It already is, but has a long way to go.

nilkn 1y ago

It's not the end of observability as we know it. However, the article also isn't totally off-base. We're almost certain to see a new agentic layer emerge and become increasingly capable for various aspects of SRE, including observability tasks like RCA. However, for this to function, most or even all of the existing observability stack will still be needed. And as long as the hallucination / reliability / trust issues with LLMs remain, human deep dives will remain part of the overall SRE work structure.

pmbauer 1y ago

It must burn a little blogging about an LLM-driven latency analysis _internal demo_ only to have Datadog launch a product in the same space a day later. https://www.datadoghq.com/blog/bits-ai-sre/ https://www.datadoghq.com/blog/bits-ai-sre/

Rastonbury 1y ago
And it wasn't just that single AI product either, they announced a bunch including an MCP server and cursor extension https://www.datadoghq.com/blog/dash-2025-new-feature-roundup-keynote/ https://www.datadoghq.com/blog/dash-2025-new-feature-roundup...

nektro 1y ago

lol

robertcary25 1y ago

[dead]