3 ms·

I believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea

• rgmerk 2y ago
  LLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.
  • taberiand 2y ago
    Set aside LLMs, why does mutation testing need to be fast? It would be fine to have mutation tests run slowly, out-of-band of the main CI pipeline. They aren't mission critical, they're smoke tests for your unit tests. Also you only need to generate a set of mutations for any particular unit once, and then again when the test code or the code under test changes.
    • rgmerk 2y ago
      Because mutation testing (should) generate a lot of mutants, which you then need to run your unit tests against.
      • taberiand 2y ago
        Yes but my point is it doesn't have to do that fast, as in unit test speeds.