3 ms·
LLM app telemetry is important, but I don’t think we have seen the right metrics yet. Nothing has convinced me that they are more useful than modern app telemet
by gdiamos 1y ago
LLM app telemetry is important, but I don’t think we have seen the right metrics yet. Nothing has convinced me that they are more useful than modern app telemetry
I don’t think tool calls or prompts or rag hits are it
That’s like saying that C++ app observability is about looking at every sys call and their arguments
Sure, if you are the OS it’s easy to instrument that, but IMO I’d rather just attach to my app and look at the logs
- jonnylaw 1y agoAttaching to the app is impractical to catch regressions in production. LLMs are probabilistic - this means you can have a regression without even changing the code / making a new deployment. A metric to alert on could be task-completion rate using LLM as a judge or synthetic tests which are run on a schedule. Then the other metrics you mentioned are useful for debugging the problem.