3 ms·

I'm working with a loop that keeps reverting and reimplementing the same code and there's not much risk of any malicious input in the context given the chat /sa

I'm working with a loop that keeps reverting and reimplementing the same code and there's not much risk of any malicious input in the context given the chat /save (which doesn't include pytest outputs it parsed for example). How to detect when this occurs? "A small number of samples can poison LLMs of any size" (yesterday) https://news.ycombinator.com/item?id=45529587 https://news.ycombinator.com/item?id=45529587

brucepro 1y ago
The version history with diffs helps with that. When you store/update memories, the system tracks changes over time. You can query the timeline to see "what solutions did I already try for X problem?" The Memory decay approach (tracking access patterns + importance decay) also surfaces this - if you keep accessing the same memory about a bug that never gets resolved, that's a signal. Not automatic loop detection yet, but the infrastructure is there. Could build a simple heuristic: if similar memory content gets created/updated N times within short timeframe, flag it as potential loop. The poisoning concern is real. Our design mitigates it somewhat - agents explicitly choose what to store (tool-based, not automatic injection), and you can prune/audit memories.
- westurner 1y ago
  > Could build a simple heuristic: if similar memory content gets created/updated N times within short timeframe, flag it as potential loop That would probably prevent waste and could scale. From https://news.ycombinator.com/context?id=45561039 https://news.ycombinator.com/context?id=45561039 : > /? llm firewall: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=llm%20firewall&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... There's probably already a name for 'Sensitivity analysis'; how does the output vary with/without particular context?

brucepro 1y ago

The version history with diffs helps with that. When you store/update memories, the system tracks changes over time. You can query the timeline to see "what solutions did I already try for X problem?" The Memory decay approach (tracking access patterns + importance decay) also surfaces this - if you keep accessing the same memory about a bug that never gets resolved, that's a signal. Not automatic loop detection yet, but the infrastructure is there. Could build a simple heuristic: if similar memory content gets created/updated N times within short timeframe, flag it as potential loop. The poisoning concern is real. Our design mitigates it somewhat - agents explicitly choose what to store (tool-based, not automatic injection), and you can prune/audit memories.

westurner 1y ago
> Could build a simple heuristic: if similar memory content gets created/updated N times within short timeframe, flag it as potential loop That would probably prevent waste and could scale. From https://news.ycombinator.com/context?id=45561039 https://news.ycombinator.com/context?id=45561039 : > /? llm firewall: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=llm%20firewall&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... There's probably already a name for 'Sensitivity analysis'; how does the output vary with/without particular context?