5 ms·
I’m convinced the magic bullet is deterministic checks. Linters, static analyzers, etc. Whatever you can do to create deterministic gates that the LLM simply mu
by rogerrogerr 3mo ago
I’m convinced the magic bullet is deterministic checks. Linters, static analyzers, etc. Whatever you can do to create deterministic gates that the LLM simply must overcome to reach a “done” state, do it. Has been making a huge difference for my team, but sister teams are so invested in writing the perfect Make No Mistakes prompt that they just can’t see it.
Basically I treat it like a junior dev. We don’t get junior devs to write code correctly by cajoling them just right, we add CI gates. It still works.
- sdesol 3mo agoWhy aren't the teams using shared checks? Are the codes in different repos?
- rogerrogerr 3mo agoThey’re very, very different projects.
- zamalek 3mo agoWouldn't have helped, sibling comment: https://news.ycombinator.com/item?id=48797883 https://news.ycombinator.com/item?id=48797883 Architectural decisions are not lintable.
- rogerrogerr 3mo agoYou can have a separate LLM as a brutal enforcer of an architecture decision. Works pretty well in the right harness.
- Toutouxc 3mo agoWider architectural guidelines, yes. "How a senior developer in this company would do it" — not so much.
- rogerrogerr 3mo agoThat’s what code review is still for.
- KronisLV 3mo ago> Architectural decisions are not lintable. I made a tool called ProjectLint to lint architectural decisions and anything that ESLint and ArchUnit and others like that wouldn't be a good fit for. It's not public yet, but is fairly rudimentary and based on Go + goja: https://github.com/dop251/goja https://github.com/dop251/goja You write rules in ESLint, it makes sure that they're followed. If it's a file in your repo, it can obviously be checked. Now whether you can describe your architecture well enough to lint it (at least the stuff you care about enforcing), that's a different question. In my case it's more like: "Oh, the AI messed up this pattern, but it will need to be followed N more times in the future, better write another projectlint rule." After a while, you can even copy them over between projects, as long as they're following the same conventions. In the same spirit of https://www.archunit.org/ https://www.archunit.org/
- cube00 3mo ago> Whatever you can do to create deterministic gates that the LLM simply must overcome to reach a “done” state, do it. First thing Gemini did when I tried that was turn off all the rules in eslint.config.mjs claiming they were "overly stylistic" Yes, it got better once I explicitly told it not to disable any rules, so I accept I was holding it wrong but I do worry just how many footguns it puts into other things because I didn't know the right guardrails to give it.
- p_kuni 3mo agoWhy does your harness allow the LLM to generate output that can disable rules & checks?
- cube00 3mo agoI have no idea because Google Antigravity is closed source.
- p_kuni 3mo agoAlthough I don't use it, Antigravity should be configurable to prevent reads / writes to certain files. You should definitely do that and ideally configure a hook to automatically run the linter after the LLM completes, then feed it back to the LLM if it's red. Don't prompt your agent to do stuff, build your loop in a deterministic manner.
- Atotalnoob 3mo agoAdd hooks to block it from being able to edit the files you don’t want changed.
- cuttothechase 3mo agoIt will burn up the tokens to get through the deterministic gates, more so when n order dependencies are involved in the mix. Enough typewritters and monkeys could get it done too.
- zmgsabst 3mo agoI found I used fewer tokens by having many short prompts than long ones — because it spent many fewer tokens thinking and narrating.