3 ms·
I'm not sure I understand the question. He's writing from the vantage point of someone with a large oTel deployment; that's the data he has to work with. Honeyc
by tptacek 1y ago
I'm not sure I understand the question. He's writing from the vantage point of someone with a large oTel deployment; that's the data he has to work with. Honeycomb has an MCP server. Instead of him clicking around Honeycomb and making inferences from the data and deciding where to drill down, an LLM did that, and found the right answer quicker than a human would have.
Where's the extra expense here? The $0.60 he spent on LLM calls with his POC agent?
- danpalmer 1y agoI think the implication is that if you have that graph you’re already half way towards a solution because you know there’s a problem. In terms of _identifying the problems_, shoving all your data into an LLM to spot irregularities would be exceptionally expensive vs traditional alerting, even though it may be much more capable at spotting potential issues without explicit alerting thresholds being set up.
- tptacek 1y agoLook, I like Honeycomb a lot and we're dependent on it for parts of our orchestrator. It's great; it accelerates investigations. But even with Honeycomb, we are sitting on an absolute mountain of telemetry data, in logs, in metrics, and in our trace indices at Honeycomb. We can solve problems by searching and drilling down into all those data sources; that's how everybody solves problems. But solving problems takes time. Just having the data in the graph does not mean we're near a solution! An LLM agent can chase down hypotheses, several at a time, and present them with collected data and plausible narratives. It'll miss things and come up with wild goose chases, but so do people during incidents.
- datadrivenangel 1y agoI think the point is that all your telemetry data is going to be very expensive.
- danpalmer 1y agoMy point, and what I think I'm clarifying from the parent comment, is that running a single starting point through an LLM to explore root causes is far cheaper than running all starting points through an LLM to find potential issues in the first place.
- nemothekid 1y agoI understand the value prop you are describing, but that is not the value prop described in the article. The article did not describe an LLM agent intuiting a hypothesis, checking with data, and coming up with a narrative all for 60 cents. The author did 80% of the work, and had the LLM do the final stretch. Maybe there's value there, but the article did not present a workflow that obviated the need for graphs. This execution gap is common in these LLM blog posts (ex. the LLM can write a login function in Swift, so the author claims LLMs will be writing production ready iOS apps tomorrow). The common thread I see is that getting from the point of "explain why this endpoint is slow" to the point of "An LLM agent can chase down hypotheses, several at a time, and present them with collected data and plausible narratives." is a lot more expensive than 60 cents - it's months of paying for an enterprise license where this feature is promised to be "just around the corner"
- tptacek 1y agoCan you be clearer about what you're defining as "80% of the work"? If it's "setting up a telemetry pipeline capable of serving an LLM agent loop", I'm going to disagree, but I don't want to disagree preemptively.
- deleted 1y ago[deleted]
- hooverd 1y agoIs the LLM making these inferences from the aether?
- techpineapple 1y agoRight, but the vision isn’t LLM troubleshooting, the title of the article isnt look at the way you can troubleshoot with an LLM, it’s “its the end of observability as we know it). he goes on to say it’s AI constantly analyzing data in a Unified sub-second database? Even without the AI that’s expensive.