3 ms·

I'm curious as to what guardrails you've tried. This is something I have been trying to get right as well. I've attempted to use lots of linting and things li

I'm curious as to what guardrails you've tried. This is something I have been trying to get right as well. I've attempted to use lots of linting and things like strong typing, duplicate checks, cyclomatic complexity, and robust tests. However, I still happen to find issues, which requires me to look at the code (at least at a high level) For example, I can say "Don't repeat yourself, and don't re-write helper functions" and I will even have a duplicate linter check, but inevitably the LLM will always want to re-write a similar yet slightly different helper function. Like it will always want to re-write something small like a trim() or a toString() function in every file.

esprehn 13d ago
Have you tried something like "Always consult the utils/ package before writing helper functions. When adding a new generic helper function justify it in your design or PR description." I have better luck telling it positive things rather than lots of "never do X" style things.
- kuczmama 12d ago
  That's a good idea to give more positive instructions as opposed to negative instructions. I think you've stated it well, I suppose the problem with negative instructions is that the LLM doesn't know what to do instead. "Never re-write a helper function" vs "Always search for helper functions before writing one" the "never... " one doesn't tell the LLM what to do, so it would have to make the logical leap from not re-writing to knowing that it should search. While it's a minor leap to make in isolation, I suppose stacking many negative rules in an AGENTS.md would assume that every time it will always make that logical conclusion on what to do.
bucket2015 13d ago
I find that if I leave an instruction in AGENTS.md to "do not do X", there's a good chance the agent will forget it. But if I add a separate post-implementation pass to "find and fix X" by the agent, it'll usually find and fix the issues. So I've started doing it for everything from naming conventions to duplicate code to other problems. It does cost more tokens, but now I get less frustrated at having to fix basic issues in the PRs.
ytoawwhra92 12d ago
> Don't repeat yourself, and don't re-write helper functions It's worth reflecting on why these things are important to you and whether they remain important in an agent-developed codebase.
- sevenseacat 12d ago
  Yes, they are both still very important for consistency throughout your codebase and any user interface for it.
  - ytoawwhra92 12d ago
    Consistency of behaviour and UI can be tested.

esprehn 13d ago

Have you tried something like "Always consult the utils/ package before writing helper functions. When adding a new generic helper function justify it in your design or PR description." I have better luck telling it positive things rather than lots of "never do X" style things.

kuczmama 12d ago
That's a good idea to give more positive instructions as opposed to negative instructions. I think you've stated it well, I suppose the problem with negative instructions is that the LLM doesn't know what to do instead. "Never re-write a helper function" vs "Always search for helper functions before writing one" the "never... " one doesn't tell the LLM what to do, so it would have to make the logical leap from not re-writing to knowing that it should search. While it's a minor leap to make in isolation, I suppose stacking many negative rules in an AGENTS.md would assume that every time it will always make that logical conclusion on what to do.

bucket2015 13d ago

I find that if I leave an instruction in AGENTS.md to "do not do X", there's a good chance the agent will forget it. But if I add a separate post-implementation pass to "find and fix X" by the agent, it'll usually find and fix the issues. So I've started doing it for everything from naming conventions to duplicate code to other problems. It does cost more tokens, but now I get less frustrated at having to fix basic issues in the PRs.

ytoawwhra92 12d ago

> Don't repeat yourself, and don't re-write helper functions It's worth reflecting on why these things are important to you and whether they remain important in an agent-developed codebase.

sevenseacat 12d ago
Yes, they are both still very important for consistency throughout your codebase and any user interface for it.
- ytoawwhra92 12d ago
  Consistency of behaviour and UI can be tested.