3 ms·
The tricky thing with git bisect run is that you usually have to have your test script not be part of your git checkout for the process to work. This is becaus
by cortesoft 2y ago
The tricky thing with git bisect run is that you usually have to have your test script not be part of your git checkout for the process to work.
This is because of the particular situation that bisect is valuable in; you find a regression that doesn’t have a test, and you aren’t sure when it was committed.
First, it has to be a regression; if it is just a bug, then there was no previous version that didn’t have the bug, it just hadn’t been found until now. It has to be something that worked before and now doesn’t.
Second, it has to not have had a test before the regression was found. If you had a test for it already, it would have been found by CI as soon as it was committed, and you wouldn’t need to figure out which commit broke it.
So if you try to add a test to the normal test suite of the code and commit it, git bisect run is not going to work; as soon as you check out the older code, your new test won’t be there and the tests will pass because your new test of the breakage doesn’t run. You have to have the new test persist across git checkouts. This is not trivial, because you can’t just exclude your test files from being updated by git bisect, since other tests will also be changing through versions. You need to have your tests always include some non-version controlled file, and you need to have added that include PRIOR to your last known good version.
The only other use case would be if you are making a lot of commits without running tests, so you actually could break a pre-existing test and not know which commit broke it. If that is your situation, you should probably change your workflow to test every commit instead of trying to get git bisect to work.
For these reasons, I have never found ‘git bisect run’ to be as valuable as it seemed when I first learned about it.
- jakub_g 2y agoAs long as your test suite just finds and runs all files in a given folder (without needing to explicitly "enable" them in some index file), this should work: - create a NEW test file in `some/path/to/test.ext` (and back it up outside repo just in case) - do NOT commit it in the repo - `git bisect` That way, bisect would check out different commits, but without touching `some/path/to/test.ext` because it's not tracked by git. It could be also helpful to make git not see some file changes through the diff/status: `git update-index --assume-unchanged path/to/some/file` to trick git into thinking the file didn't change. (Although when you checkout a commit which did modify that file, this would crash).
- hughdbrown 2y agoThis is exactly what I do: I put my test file outside the git repo's tree. This does not introduce any complications to testing and is minimally annoying.
- chw9e 2y agoNow with AI test frameworks like stagehand it’s actually possible to write end to end tests after a bug appears that can be backwards compatible as long as changes to the dom are not too extreme. But things like broken selectors won’t be an issue. I wrote about that here: https://qckfx.com/blog/ai-powered-stagehand-git-bisect-finding-and-fixing-the-commit-that-broke-your-code https://qckfx.com/blog/ai-powered-stagehand-git-bisect-findi...
- chw9e 2y agoAnother good case is for rolling back a single bad commit from a batch that got merged into main at the same time. Doing batch merges with a merge queue can speed up things if you have a ton of longer running end to end and integration tests. But then if a test fails you need to identify which commit out of the batch is causing it so you don’t reject the entire batch.
- arccy 2y agogit bisect run can take a script, you can easily script adding a test case and running the test, either to an existing file or a new file.