3 ms·
All very good questions. Agents are actively destroy QA gates in many ways, usually by cheating ("the test is buggy, not my changes" — changes the test), or ju
by aka-rider 2mo ago
All very good questions.
Agents are actively destroy QA gates in many ways, usually by cheating ("the test is buggy, not my changes" — changes the test), or just rot QA slowly by writing buggy overcomplicated tests
What works for me 10/10 is fuzzing and my own constant usage. For this project specifically (text editor), I asked LLM to create human-like fuzzing session, it sends keystrokes like: "the user is searching for a file, editing, <ordering a lizard>, saves changes".
On top of it, I run https://mutants.rs/ https://mutants.rs/ which is kind of tests fuzzing. It flips random switches in the app itself, and if tests are silent - they missed a bug.
The downside of this, is I usually find bugs after 1-2 hours of running.
I use local Qwen to babysit these sessions, to make initial investigation, a repro case, and file a ticket.
- metaltyphoon 2mo agoWhy are you just pasting LLM answers :(? I see this constantly in Slack DMs to every day from work. It hurts
- aka-rider 2mo agoThis is genuinely how I write :'( It is probably because I read tons and tons of LLM output.
- tensegrist 2mo agothis is not llm writing. there's no need to startle at the sight of an em-dash