4 ms·

+1 At Meta we do similar heuristics on which tests to run per PR. When the system gets it wrong, which is often, its painful and leads to merged code that brok

• unshavedyak 1y ago
  Fwiw i feel like you could still do both, no? Ie run all tests daily or whatever cadence is not painful but still protective, but _also_ use heuristics to improve PR responsiveness. Like anything it's about tradeoffs. Though if it were me i'd simply write a mechanism which deterministically decides which areas of code pertain to which tests and use that simple program to determine which tests run. The algorithm could be loosely as simple as things like code owners and git-blame, but relative to some big set of Code->Test that you can have Claude/etc build ahead of time. The difference being it's deterministic between PRs and can be audited by humans for anything obviously poor or w/e. As much as LLMs are interesting to me i hate using them in places where i want consistency. CI seems terrible for them.. to me at least.
  • sevazhidkov 1y ago
    To be fair, the most interesting bugs that can be caught with tests always feel like “I would’ve never guessed that part of system actually depends on this one”.
• andsoitis 1y ago
  > When the system gets it wrong, which is often Which suggests there might be big room for improvement. If LLMs open up new ways to be more precise about which E2E to run, that is a significant benefit, because you’d manage risk but reduce the time and resources it takes.