28 ms·
I've tested many flows involving linters. Results are far from ideal - agents tend to work around linters, mass-add ignore annotations, etc, especially in situa
by pshirshov 5mo ago
I've tested many flows involving linters. Results are far from ideal - agents tend to work around linters, mass-add ignore annotations, etc, especially in situations when fixing one warning/error triggers another (and that happens regularly).
- the_af 5mo agoWhat if we tell the agent NOT to add ignore annotations (or to ask about them if there's no other reasonable way to proceed)?
- pshirshov 5mo agoThey tend to ignore such instructions on first circular issue - even with Opus you have to kick it really hard, insist on generalization and intervene manually. In my opinion this is not a productive/workable approach for large projects. Typical failure mode: "I fix pyright error A, it causes pyright error B, pyright is broken, I will exclude both A and B through pyright config and will add ignore annotations for both A and B and will write a couple of idiotic comments about that".
- codybontecou 5mo agoYou're devolving into the realm of "What if we tell the agent to just get it right?" Relying on the prompt to ensure the code it writes is correct is where things fail. Types, tests, linting, etc. are deterministic tools the agents tend to respect.
- jimbokun 5mo agoLLMs really are just like an intern in their first coding job!