3 ms·
You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I
by anitil 28d ago
You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.
- ngruhn 28d agoI thought so too but it's probably because I never review the tests. I made an exception recently and found tons of assert(CONSTANT_CONFIG == valueOfConfig) or tests for keywords in prompts: assert(prompt.includes("repo url"))
- genxy 26d agoWas your prompt, "make lazy ineffectual tests?" Tests are code as well, they take the same effort to create, if not more. You need to specify the properties you want to test.
- __alexs 27d agoI don't think these results show that. I think they show that additional prompting for testing approaches have wide differences in error rate (worst has twice the error rate of the best) but actually no extra prompting is pretty fine and most custom prompts are worse than no prompt.
- ponector 27d agoMost developers are not good at testing. That's why the role of test engineer exists. I've seen in multiple projects things like assertTrue(true). I'm sure the agent is better in testing than average enterprise developer.
- zaphirplane 27d agobig tech is moving / has moved away from the role.
- ponector 27d agoUnfortunately for the users, testing has been pushed to them. You can deploy whatever slop you want - nobody cares. More lines of code!
- simonwsimonw 27d ago[dead]
- gmueckl 27d agoIsn't that just a form of Dunning-Kruger effect related to the parts that person is bad in? It looks good because they don't have the depth of knowledge/expertise to see the deficiencies.