3 ms·
This is one of the most practical breakdowns I’ve seen for a while. The spec.md as a living architecture map is smart, and documenting auth guard pattern sites
by jfaganel99 6mo ago
This is one of the most practical breakdowns I’ve seen for a while. The spec.md as a living architecture map is smart, and documenting auth guard pattern sites as new modules get added is exactly the kind of thing that prevents issues creeping in.
The bit I’d push on: do your reviewer agents catch logic errors… things like a double negative auth check or a race condition in a payment flow. Those usually pass a check because code looks intentional and clean. Curious whether your reviewers are prompted specifically for security logic or more for spec conformance?
“Don’t merge code you don’t understand” is the right closer. Most setups don’t force that discipline cause people dont have the knowledge :)
- fhouser 6mo agoOpus 4.6 usually doesn't disappoint .. No double negative auth checks or race conditions to report on, but I can say that introducing new functionality and patterns mostly requires a few cycles before the "repeatable pattern" is cleanly documented in the spec. When bugs do come up, the agent is quite good at finding the root cause and implementing a fix.
- jfaganel99 6mo agoWorking on a model benchmark focused on which model is good for these tasks. Keep you posted
- fhouser 6mo agoThanks,that would be great.
- jfaganel99 6mo agoAs promised here is the open-source GitRepo so you can give it a go with your tooling: https://github.com/kolega-ai/Real-Vuln-Benchmark https://github.com/kolega-ai/Real-Vuln-Benchmark Updated benchmark results published here also. BTW, with v002 we are consistently hitting 75+