3 ms·

LLMs won't replace observability, but they absolutely change the game. Asking "why is latency spiking" and getting a coherent root cause in seconds is powerful.

LLMs won't replace observability, but they absolutely change the game. Asking "why is latency spiking" and getting a coherent root cause in seconds is powerful. You still need good telemetry, but this shifts the value from visualizing data to explaining it.

Nathanba 1y ago
I was initially agreeing with the article but it's a clever marketing piece. Nothing changes, these graphs were already really easy to read and if they weren't then they should be. You should already be capable of zooming into your latency spike within seconds and clicking on it and seeing which method was slow. So asking the AI will be more comfortable but it doesn't change anything.
ActorNightly 1y ago
LLMS are basically just like higher level programming tools - knowing how to utilize them is the key. Best practice is not to depend on them for correctness, but instead utilize them as automatic maps from data->action that you would otherwise have to write manually. For example, I wrote my own MCP server in Python that basically makes it easy to record web browser activities and replay them using playwright. When I have to look at logs or inspect metrics, I record the workflow, and import it as an MCP tool. Then I keep a prompt file where I record what the task was, tool name, description of the output, and what is the final answer given an output. So now, instead of doing the steps manually, I basically just ask Claude to do things. At some point, I am going to integrate real time voice recording and trigger on "Hey Claude" so I don't even have to type. The only thing I wish someone would do is basically make a much smaller model with limited training only on things related to computer science, so it can run at high resolution on a single Nvidia card with fast inference.
jacobsenscott 1y ago
The problem with LLMs is the answer always sounds right, no matter if it is or isn't. If you already know the answer to a question it is kind of fun to see an LLM get lucky and cobble together a correct answer. But they are otherwise useless - you need to do all the same work you would do anyway to check the LLM's "answer".
- dcre 1y ago
  There’s a world of difference between “always sounds right, but actually is right 80% of the time” and “always sounds right, but actually is right 99% of the time.” It has seemed clear to me for a while that we are on the way to the latter though a combination of model improvement and boring, straightforward engineering on scaffolding (e.g., spending additional compute verifying answers by trying to produce counterarguments). Model improvement is maybe less straightforward but the trajectory is undeniable and showing no sign of plateauing.

Nathanba 1y ago

I was initially agreeing with the article but it's a clever marketing piece. Nothing changes, these graphs were already really easy to read and if they weren't then they should be. You should already be capable of zooming into your latency spike within seconds and clicking on it and seeing which method was slow. So asking the AI will be more comfortable but it doesn't change anything.

ActorNightly 1y ago

LLMS are basically just like higher level programming tools - knowing how to utilize them is the key. Best practice is not to depend on them for correctness, but instead utilize them as automatic maps from data->action that you would otherwise have to write manually. For example, I wrote my own MCP server in Python that basically makes it easy to record web browser activities and replay them using playwright. When I have to look at logs or inspect metrics, I record the workflow, and import it as an MCP tool. Then I keep a prompt file where I record what the task was, tool name, description of the output, and what is the final answer given an output. So now, instead of doing the steps manually, I basically just ask Claude to do things. At some point, I am going to integrate real time voice recording and trigger on "Hey Claude" so I don't even have to type. The only thing I wish someone would do is basically make a much smaller model with limited training only on things related to computer science, so it can run at high resolution on a single Nvidia card with fast inference.

jacobsenscott 1y ago

The problem with LLMs is the answer always sounds right, no matter if it is or isn't. If you already know the answer to a question it is kind of fun to see an LLM get lucky and cobble together a correct answer. But they are otherwise useless - you need to do all the same work you would do anyway to check the LLM's "answer".

dcre 1y ago
There’s a world of difference between “always sounds right, but actually is right 80% of the time” and “always sounds right, but actually is right 99% of the time.” It has seemed clear to me for a while that we are on the way to the latter though a combination of model improvement and boring, straightforward engineering on scaffolding (e.g., spending additional compute verifying answers by trying to produce counterarguments). Model improvement is maybe less straightforward but the trajectory is undeniable and showing no sign of plateauing.