3 ms·

You didn't reduce e2e test time. You've reduced the e2e test coverage by only running those tests that the LLM tells you to run.

un_montagnard 1y ago
Reminds me that time when I was doing some POC on an existing code base and I reduced build time by deleting all unit tests
sghiassy 1y ago
+1 At Meta we do similar heuristics on which tests to run per PR. When the system gets it wrong, which is often, its painful and leads to merged code that broke unrun tests
- unshavedyak 1y ago
  Fwiw i feel like you could still do both, no? Ie run all tests daily or whatever cadence is not painful but still protective, but _also_ use heuristics to improve PR responsiveness. Like anything it's about tradeoffs. Though if it were me i'd simply write a mechanism which deterministically decides which areas of code pertain to which tests and use that simple program to determine which tests run. The algorithm could be loosely as simple as things like code owners and git-blame, but relative to some big set of Code->Test that you can have Claude/etc build ahead of time. The difference being it's deterministic between PRs and can be audited by humans for anything obviously poor or w/e. As much as LLMs are interesting to me i hate using them in places where i want consistency. CI seems terrible for them.. to me at least.
  - sevazhidkov 1y ago
    To be fair, the most interesting bugs that can be caught with tests always feel like “I would’ve never guessed that part of system actually depends on this one”.
- andsoitis 1y ago
  > When the system gets it wrong, which is often Which suggests there might be big room for improvement. If LLMs open up new ways to be more precise about which E2E to run, that is a significant benefit, because you’d manage risk but reduce the time and resources it takes.
manojlds 1y ago
Why do you respond to the title when that's addressed in the opening.
- CodeMage 1y ago
  No, it's not. The key concept here is the reduction of coverage. The opening part of the post makes certain claims -- "catch bugs before they ship" and "keep our master branch always clean" -- that can only be true if the coverage is not reduced. But, as other replies point out, if the heuristic for test selection has false negatives, the coverage is effectively reduced, which can result in bugs. That problem can be bad enough when the heuristic is deterministic. AI throws determinism out of the window, which only exacerbates the problem.
spaceywilly 1y ago
Yeah I’m not seeing any evidence that this actually works. I would’ve like to see some testing where they intentionally introduce a bug (ideally a tricky bug in a part of code that isn’t directly changed by the diff) and see if Claude catches it. A good middle ground could be to allow the diff to land once the “AI quick check” passed, then keep the full test suite running in the background. If they run them side by side for a while and see that the AI quick check caught the failing test every time, I’d be convinced.

un_montagnard 1y ago

Reminds me that time when I was doing some POC on an existing code base and I reduced build time by deleting all unit tests

sghiassy 1y ago

+1 At Meta we do similar heuristics on which tests to run per PR. When the system gets it wrong, which is often, its painful and leads to merged code that broke unrun tests

unshavedyak 1y ago
Fwiw i feel like you could still do both, no? Ie run all tests daily or whatever cadence is not painful but still protective, but _also_ use heuristics to improve PR responsiveness. Like anything it's about tradeoffs. Though if it were me i'd simply write a mechanism which deterministically decides which areas of code pertain to which tests and use that simple program to determine which tests run. The algorithm could be loosely as simple as things like code owners and git-blame, but relative to some big set of Code->Test that you can have Claude/etc build ahead of time. The difference being it's deterministic between PRs and can be audited by humans for anything obviously poor or w/e. As much as LLMs are interesting to me i hate using them in places where i want consistency. CI seems terrible for them.. to me at least.
- sevazhidkov 1y ago
  To be fair, the most interesting bugs that can be caught with tests always feel like “I would’ve never guessed that part of system actually depends on this one”.
andsoitis 1y ago
> When the system gets it wrong, which is often Which suggests there might be big room for improvement. If LLMs open up new ways to be more precise about which E2E to run, that is a significant benefit, because you’d manage risk but reduce the time and resources it takes.

manojlds 1y ago

Why do you respond to the title when that's addressed in the opening.

CodeMage 1y ago
No, it's not. The key concept here is the reduction of coverage. The opening part of the post makes certain claims -- "catch bugs before they ship" and "keep our master branch always clean" -- that can only be true if the coverage is not reduced. But, as other replies point out, if the heuristic for test selection has false negatives, the coverage is effectively reduced, which can result in bugs. That problem can be bad enough when the heuristic is deterministic. AI throws determinism out of the window, which only exacerbates the problem.

spaceywilly 1y ago

Yeah I’m not seeing any evidence that this actually works. I would’ve like to see some testing where they intentionally introduce a bug (ideally a tricky bug in a part of code that isn’t directly changed by the diff) and see if Claude catches it. A good middle ground could be to allow the diff to land once the “AI quick check” passed, then keep the full test suite running in the background. If they run them side by side for a while and see that the AI quick check caught the failing test every time, I’d be convinced.