5 ms·

LLM generated tests in my experience are really poor

• redox99 9mo ago
  Doesn't change the fact that what I mentioned greatly improves agent accuracy.
  • Madmallard 9mo ago
    [flagged]
  • dns_snek 9mo ago
    AI-generated implementation with AI-generated tests left me with some of the worst code I've witnessed in my life. Many of the passing tests it generated were tautologies (i.e. they would never fail even if behavior was incorrect). When the tests failed the agent tended to change the (previously correct) test making it pass but functionally incorrect, or it "wisely" concluded that both the implementation and the test are correct but that there are external factors making the test fail (there weren't). It behaved much like a really naive junior.
    • simonw 9mo ago
      Which coding agent and which model?
    • cindyllm 9mo ago
      [dead]