2 ms·

i would love to used to use it change code in ways that compiles and see if test fails. Coverage metric sometimes doesn't really tell you if some piece of code

• sesm 2y ago
  Coverage metric can tell if lines of code were executed, but they can't tell if execution result was checked.
• taberiand 2y ago
  I believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea
  • rgmerk 2y ago
    LLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.
    • taberiand 2y ago
      Set aside LLMs, why does mutation testing need to be fast? It would be fine to have mutation tests run slowly, out-of-band of the main CI pipeline. They aren't mission critical, they're smoke tests for your unit tests. Also you only need to generate a set of mutations for any particular unit once, and then again when the test code or the code under test changes.
      • rgmerk 2y ago
        Because mutation testing (should) generate a lot of mutants, which you then need to run your unit tests against.
        • taberiand 2y ago
          Yes but my point is it doesn't have to do that fast, as in unit test speeds.