3 ms·

I think the implication is that if you have that graph you’re already half way towards a solution because you know there’s a problem. In terms of _identifying

I think the implication is that if you have that graph you’re already half way towards a solution because you know there’s a problem. In terms of _identifying the problems_, shoving all your data into an LLM to spot irregularities would be exceptionally expensive vs traditional alerting, even though it may be much more capable at spotting potential issues without explicit alerting thresholds being set up.

tptacek 1y ago
Look, I like Honeycomb a lot and we're dependent on it for parts of our orchestrator. It's great; it accelerates investigations. But even with Honeycomb, we are sitting on an absolute mountain of telemetry data, in logs, in metrics, and in our trace indices at Honeycomb. We can solve problems by searching and drilling down into all those data sources; that's how everybody solves problems. But solving problems takes time. Just having the data in the graph does not mean we're near a solution! An LLM agent can chase down hypotheses, several at a time, and present them with collected data and plausible narratives. It'll miss things and come up with wild goose chases, but so do people during incidents.
- datadrivenangel 1y ago
  I think the point is that all your telemetry data is going to be very expensive.
  - danpalmer 1y ago
    My point, and what I think I'm clarifying from the parent comment, is that running a single starting point through an LLM to explore root causes is far cheaper than running all starting points through an LLM to find potential issues in the first place.
- nemothekid 1y ago
  I understand the value prop you are describing, but that is not the value prop described in the article. The article did not describe an LLM agent intuiting a hypothesis, checking with data, and coming up with a narrative all for 60 cents. The author did 80% of the work, and had the LLM do the final stretch. Maybe there's value there, but the article did not present a workflow that obviated the need for graphs. This execution gap is common in these LLM blog posts (ex. the LLM can write a login function in Swift, so the author claims LLMs will be writing production ready iOS apps tomorrow). The common thread I see is that getting from the point of "explain why this endpoint is slow" to the point of "An LLM agent can chase down hypotheses, several at a time, and present them with collected data and plausible narratives." is a lot more expensive than 60 cents - it's months of paying for an enterprise license where this feature is promised to be "just around the corner"
  - tptacek 1y ago
    Can you be clearer about what you're defining as "80% of the work"? If it's "setting up a telemetry pipeline capable of serving an LLM agent loop", I'm going to disagree, but I don't want to disagree preemptively.
    - deleted 1y ago
      [deleted]

tptacek 1y ago

Look, I like Honeycomb a lot and we're dependent on it for parts of our orchestrator. It's great; it accelerates investigations. But even with Honeycomb, we are sitting on an absolute mountain of telemetry data, in logs, in metrics, and in our trace indices at Honeycomb. We can solve problems by searching and drilling down into all those data sources; that's how everybody solves problems. But solving problems takes time. Just having the data in the graph does not mean we're near a solution! An LLM agent can chase down hypotheses, several at a time, and present them with collected data and plausible narratives. It'll miss things and come up with wild goose chases, but so do people during incidents.

datadrivenangel 1y ago
I think the point is that all your telemetry data is going to be very expensive.
- danpalmer 1y ago
  My point, and what I think I'm clarifying from the parent comment, is that running a single starting point through an LLM to explore root causes is far cheaper than running all starting points through an LLM to find potential issues in the first place.
nemothekid 1y ago
I understand the value prop you are describing, but that is not the value prop described in the article. The article did not describe an LLM agent intuiting a hypothesis, checking with data, and coming up with a narrative all for 60 cents. The author did 80% of the work, and had the LLM do the final stretch. Maybe there's value there, but the article did not present a workflow that obviated the need for graphs. This execution gap is common in these LLM blog posts (ex. the LLM can write a login function in Swift, so the author claims LLMs will be writing production ready iOS apps tomorrow). The common thread I see is that getting from the point of "explain why this endpoint is slow" to the point of "An LLM agent can chase down hypotheses, several at a time, and present them with collected data and plausible narratives." is a lot more expensive than 60 cents - it's months of paying for an enterprise license where this feature is promised to be "just around the corner"
- tptacek 1y ago
  Can you be clearer about what you're defining as "80% of the work"? If it's "setting up a telemetry pipeline capable of serving an LLM agent loop", I'm going to disagree, but I don't want to disagree preemptively.
  - deleted 1y ago
    [deleted]