3 ms·

> Yes, and I'm not exaggerating. Claude never missed a relevant E2E test. I was waiting to see this part demonstrated and validated - for a given pr, whether y

• jampa 1y ago
  I should have added that to the post, but we verify by running a full test suite in a cron job, so if a bug slips, we will know. The full suite sometimes breaks, but these are due to changes outside of PR (e.g., a vendor goes down).
  • kelnos 1y ago
    That doesn't verify what you think it does, though. It doesn't verify that the LLM ran all the tests necessary to exercise the code paths that were changed. "Running all necessary tests" is indistinguishable from "we did a good job on these PRs and didn't write any new bugs that would have been caught by the tests". This is the classic situation where you can prove a negative (if all the LLM-selected tests pass, but there's a new failure in the full suite), but can't prove a positive (if all the LLM-selected tests pass, and the full suite also passes, it could just mean there were no new bugs, not that the LLM selected the right tests). The only way to verify that is to do what GP suggested: for each PR, manually make a list of the tests that you believe should be run, and then compare that to the list of tests that the LLM actually runs. Obviously you aren't going to do this forever, but you should either do it for every single PR for some number of PRs, or pick a representative sample over time instead.