3 ms·
Context pollution and rot are probably more important than memory, because facts can usually be retrieved if the agent is good at following breadcrumbs. What's
by nullbio 1mo ago
Context pollution and rot are probably more important than memory, because facts can usually be retrieved if the agent is good at following breadcrumbs.
What's also the biggest killer is code rot. Agents are particularly good at death by thousand cuts. They implement something poorly, or incorrectly, or introduce a bad pattern into the project. Then they continue to amplify that badness over time, as they continue to copy from it on subsequent work. It spreads like a virus.
Keeping these seeds out of the project is very difficult, and cleaning up the rot is very difficult. It also seems like a hard problem to solve because following the existing codebase is something that is good when the code is good, but bad when it is bad. So, seemingly, the solution means more thinking and evaluation for every change that is being made.
- gdad 1mo agoTruly. Doing this for coding agents is an interesting and different shaped problem.
- ShinyLeftPad 1mo ago> They implement something poorly, or incorrectly, or introduce a bad pattern into the project. Then they continue to amplify that badness over time, as they continue to copy from it on subsequent work. It spreads like a virus. > Keeping these seeds out of the project is very difficult, and cleaning up the rot is very difficult. My "aha" moment was when I realized this goes for all spheres of life where this tech is/will be introduced.
- taneq 1mo agoIt goes for all spheres of life, full stop. I’m not sure if agents struggle with this because they learned it from humans, or if they struggle with it because it’s a universally challenging problem, but it’s something we share with them.
- ShinyLeftPad 1mo agoThe comment highlighted how LLMs exacerbate the issue by entrenching the preexisting issues.
- Terretta 1mo agoIndeed, and reply to comment seemed a "yes and" -- As with humans. It's curious how much of these could apply to either: https://en.wikipedia.org/wiki/Reconstructive_memory https://en.wikipedia.org/wiki/Reconstructive_memory https://en.wikipedia.org/wiki/Misinformation_effect https://en.wikipedia.org/wiki/Misinformation_effect And many mechanisms exacerbate issues by entrenching preexisting issues.
- ShinyLeftPad 1mo agoI don't understand. The comment said if humans want to change the route, this tech makes it more difficult. Human inertia is X, inertia with this tech is X ^ Y. The Y is the issue being discussed.
- Schlagbohrer 1mo ago"Out of the crooked timber of humanity, no straight thing was ever wrought"
- Terr_ 1mo agoYou full-stopped too soon: It may happen everywhere, but it doesn't happen the same way everywhere or for the same reasons. LLMs will create different kinds of corruption than humans, because the underlying mechanisms are different. Our ability to manage type of corruption will depend on how whether they can be predicted by math or intuition.
- taneq 1mo agoYou’re kind of begging the question that I asked after the full stop. I was wondering how much the underlying mechanism matters if it’s modelling the same environment to a similar level of effectiveness.
- Terr_ 1mo ago> Then they continue to amplify that badness over time Also, with "self-bias", models are also likely to grow new content into spots that match their subtle fingerprints from the past. That might come at the expense of whatever corrected "we should avoid that and do this instead" alternative some human added for future architecture.
- MacketSWE 1mo ago[flagged]
- fsiefken 1mo agoYes so regular human and agentic evaluation of the coding agent output, scoring it on specific criteria? https://github.com/harness/harness-evals https://github.com/harness/harness-evals