3 ms·

I feel like the alternate title of this could be “how to 10x your observability costs with this one easy trick”. It didn’t really show a way to get rid of all t

I feel like the alternate title of this could be “how to 10x your observability costs with this one easy trick”. It didn’t really show a way to get rid of all the graphs, the prompt was “show me why my latency spikes every four hours”. That’s really cool, but in order to generate that prompt you need alerts and graphs. How do you know you’re latency is spiking to generate the prompt? The devil seems to be in the details, but you’re running a whole bunch more compute for anomaly detection and “ Sub-second query performance, unified data storage”, which again sounds like throwing enormous amounts of more money at the problem. I can totally see why this is great for honeycomb though, they’re going to make bank.

techpineapple 1y ago
Additionally, I wonder if any of this fixes the fact that anomaly detection in alerting is traditionally a really hard problem, and one I’ve hardly seen done well. Of any set of packaged or recommended alerts, I probably only use 1% of them because anomalies are often the norm.
- tptacek 1y ago
  That reinforces his argument.
  - techpineapple 1y ago
    If he stopped at “LLms can help rca’s, I’d agree with you.
- zer00eyz 1y ago
  > because anomalies are often the norm You fix these issues or you tune your alert system to make it clear that they aren't actionable. Otherwise you end up turning them off so your system doesn't turn into the boy who cried wolf (or worse teams learn to ignore it and it becomes useless) Bayesian filters, and basic dirivative functions (think math) can do a lot to tame output from these systems. These arent "product features" so in most orgs they dont get the attention they need or deserve.
  - zdragnar 1y ago
    > or worse teams learn to ignore it and it becomes useless This is basically every team I've worked with. Product wants new features, and doesn't want to spend on existing features. Hurry up and write new stuff! Ignore problems and they'll go away! Also: I've already reported this bug! Why haven't the developers fixed it is yet?
tptacek 1y ago
I'm not sure I understand the question. He's writing from the vantage point of someone with a large oTel deployment; that's the data he has to work with. Honeycomb has an MCP server. Instead of him clicking around Honeycomb and making inferences from the data and deciding where to drill down, an LLM did that, and found the right answer quicker than a human would have. Where's the extra expense here? The $0.60 he spent on LLM calls with his POC agent?
- danpalmer 1y ago
  I think the implication is that if you have that graph you’re already half way towards a solution because you know there’s a problem. In terms of _identifying the problems_, shoving all your data into an LLM to spot irregularities would be exceptionally expensive vs traditional alerting, even though it may be much more capable at spotting potential issues without explicit alerting thresholds being set up.
  - tptacek 1y ago
    Look, I like Honeycomb a lot and we're dependent on it for parts of our orchestrator. It's great; it accelerates investigations. But even with Honeycomb, we are sitting on an absolute mountain of telemetry data, in logs, in metrics, and in our trace indices at Honeycomb. We can solve problems by searching and drilling down into all those data sources; that's how everybody solves problems. But solving problems takes time. Just having the data in the graph does not mean we're near a solution! An LLM agent can chase down hypotheses, several at a time, and present them with collected data and plausible narratives. It'll miss things and come up with wild goose chases, but so do people during incidents.
    - datadrivenangel 1y ago
      I think the point is that all your telemetry data is going to be very expensive.
      - danpalmer 1y ago
zer00eyz 1y ago
Also we need to talk about what should be logged and where. There seem to be two schools of thought, just enough to tell something is wrong but not what it is - OR - you get to drink from the firehose. And most orgs go from the first to the second. As to where, well thats at the hardware/vm/container level, and mirror and extend what it does. Nothing worse than 20 different ideas of how to log and rotate and trying to figure out who did what, when where and why. If you can't match a log entry to a running environment... well. I weep quietly inside when some or all of this goes through one, or several S3 buckets for no good reason.

techpineapple 1y ago

Additionally, I wonder if any of this fixes the fact that anomaly detection in alerting is traditionally a really hard problem, and one I’ve hardly seen done well. Of any set of packaged or recommended alerts, I probably only use 1% of them because anomalies are often the norm.

tptacek 1y ago
That reinforces his argument.
- techpineapple 1y ago
  If he stopped at “LLms can help rca’s, I’d agree with you.
zer00eyz 1y ago
> because anomalies are often the norm You fix these issues or you tune your alert system to make it clear that they aren't actionable. Otherwise you end up turning them off so your system doesn't turn into the boy who cried wolf (or worse teams learn to ignore it and it becomes useless) Bayesian filters, and basic dirivative functions (think math) can do a lot to tame output from these systems. These arent "product features" so in most orgs they dont get the attention they need or deserve.
- zdragnar 1y ago
  > or worse teams learn to ignore it and it becomes useless This is basically every team I've worked with. Product wants new features, and doesn't want to spend on existing features. Hurry up and write new stuff! Ignore problems and they'll go away! Also: I've already reported this bug! Why haven't the developers fixed it is yet?

tptacek 1y ago

I'm not sure I understand the question. He's writing from the vantage point of someone with a large oTel deployment; that's the data he has to work with. Honeycomb has an MCP server. Instead of him clicking around Honeycomb and making inferences from the data and deciding where to drill down, an LLM did that, and found the right answer quicker than a human would have. Where's the extra expense here? The $0.60 he spent on LLM calls with his POC agent?

danpalmer 1y ago
I think the implication is that if you have that graph you’re already half way towards a solution because you know there’s a problem. In terms of _identifying the problems_, shoving all your data into an LLM to spot irregularities would be exceptionally expensive vs traditional alerting, even though it may be much more capable at spotting potential issues without explicit alerting thresholds being set up.
- tptacek 1y ago
  Look, I like Honeycomb a lot and we're dependent on it for parts of our orchestrator. It's great; it accelerates investigations. But even with Honeycomb, we are sitting on an absolute mountain of telemetry data, in logs, in metrics, and in our trace indices at Honeycomb. We can solve problems by searching and drilling down into all those data sources; that's how everybody solves problems. But solving problems takes time. Just having the data in the graph does not mean we're near a solution! An LLM agent can chase down hypotheses, several at a time, and present them with collected data and plausible narratives. It'll miss things and come up with wild goose chases, but so do people during incidents.
  - datadrivenangel 1y ago
    I think the point is that all your telemetry data is going to be very expensive.
    - danpalmer 1y ago

zer00eyz 1y ago

Also we need to talk about what should be logged and where. There seem to be two schools of thought, just enough to tell something is wrong but not what it is - OR - you get to drink from the firehose. And most orgs go from the first to the second. As to where, well thats at the hardware/vm/container level, and mirror and extend what it does. Nothing worse than 20 different ideas of how to log and rotate and trying to figure out who did what, when where and why. If you can't match a log entry to a running environment... well. I weep quietly inside when some or all of this goes through one, or several S3 buckets for no good reason.