3 ms·
What I think we will see in the future is company-wide analysis of anonymised communications with agents, and derivations of common pain points and themes based
by LudwigNagasena 7mo ago
What I think we will see in the future is company-wide analysis of anonymised communications with agents, and derivations of common pain points and themes based on that.
Ie, the derivation of “knowledge units” will be passive. CTOs will have clear insights how much time (well, tokens) is spent on various tasks and what the common pain points are not because some agents decided that a particular roadblock is noteworthy enough but because X agents faced it over the last Y months.
- layer8 7mo agoHow will you derive pain points and roadblocks if you don’t trust LLMs to identify them?
- LudwigNagasena 7mo agoI trust that an LLM can fix a problem without the help of other agents that are barely different from it. What it lacks is the context to identify which problems are systemic and the means to fix systemic problems. For that you need aggregate data processing.
- layer8 7mo agoWhat I mean is, how do you identify a “problem” in the first place?
- LudwigNagasena 7mo agoYou analyze each conversation with an LLM: summarize it, add tags, identify problematic tools, etc. The metrics go to management, some docs are auto-generated and added to the company knowledge base like all other company docs. It’s like what they do in support or sales. They have conversational data and they use it to improve processes. Now it’s possible with code without any sort of proactive inquiry from chatbots.
- layer8 7mo agoWho is “you” in the first sentence? A human or an LLM? It seems to me that only the latter would be practical, given the volume. But then I don’t understand how you trust it to identify the problems, while simultaneously not trusting LLMs to identify pain points and roadblocks.
- LudwigNagasena 7mo agoAn LLM. A coding LLM writes code with its tools for writing files, searching docs, reading skills for specific technologies and so on; and the analysis LLM processes all interactions, summarizes them, tags issues, tracks token use for various task types, and identifies patterns across many sessions.
- cyanydeez 7mo agooh man, can youimagine having this much faith in a statistical model that can be torpedo'd cause it doesn't differentiate consistently between a template, a command, and an instruction?
- ray_v 7mo agoBetter question yet, how do you have agents contribute openly without an insane risk of leaking keys, credentials, PII, etc, etc? Again it's a terrible idea, and yet I'll SMASH that like button and use it anyway