7 ms·
You say you have to rely on humans and heuristics to determine which CL to revert. Couldn't you automatically re-run just the failing tests on the batch? More
by Ecco 11y ago
You say you have to rely on humans and heuristics to determine which CL to revert. Couldn't you automatically re-run just the failing tests on the batch?
More importantly, you seem to be running all the tests, as often as you can. A commit changing a text label shouldn't require running all the tests, that just doesn't scale. Have you consider splitting Chromium into different projects, where commits for one project would result in the tests of that project, and only that project?
- phasmantistes 11y agoWe actually are working on systems which re-run the failing tests on all CLs within a failing batch. Unfortunately, we haven't been able to do this previously due to resource constraints. In particular, if a batch had 10 commits in it, you can either run 10 additional compiles and tests (didn't have the resources), or you can binary-search/bisect to find the failure (takes too long). It is often faster to use a heuristic to say "Test_Foo failed, and Foo_Src was modified in change X, so revert change X". We don't like relying on heuristics, but until recently it was our only choice. We actually don't run all the tests all the time. We have a good dependency system (called Gyp/GN, similar to Bazel) which knows which tests rely on which source files. When the infrastructure compiles a new change, it only runs the tests which depend on source files touched by that change.