5 ms·

Recently git bisect gained got a new capability that was mysteriously missing before: bisect —first-parent. With that, it became possible to bisect also in repo

Recently git bisect gained got a new capability that was mysteriously missing before: bisect —first-parent. With that, it became possible to bisect also in repos where all commits on feature branches may not be buildable or passing tests, but commits on the development/main/integration branch does. A feature I’m still missing is to bisect using a new test. That is, I have discovered a bug in the program that there is no test for, so I author a failing test. Now I want to find out when this failure occurred but I can’t bisect because even the previous commit is green as the test didn’t exist. It’s possible to work around by placing the test outside the repo root and bisecting with a script that “authors” the test (e.g copies the file) at each step, but it’s cumbersome and feels like something that could be possible to add to the bisect command, e.g an option to merge a commit (the new test) before testing each step of the bisect.

k_ 4y ago
This might be a good argument for keeping tests as a separate repository. Which would bring a lot more problems of its own, though...
OJFord 4y ago
Just put the test in a new untracked file?
- alkonaut 4y ago
  That works but only for languages/build-systems where the mere presence of the test is enough to make it run. In others you might have xml project files, test lists etc which makes the addition of a test require a modification of existing files.
  - Izkata 4y ago
    The test subcommand of bisect can do anything you want, so it's easy to make a wrapper that does that prep work before starting that actual tests. Just have to remember to pass back the tests' exit code after cleaning up the repository for git.
  - OJFord 4y ago
    Good point, though I don't like the idea of a setup where your test file's only executed if you remember to do that personally (if the tooling doesn't already do similar, I'd at least want to generate the list you describe by globbing the relevantly named/located files). You could just copy that entrypoint into another untracked file, have its list include (or be only - quicker) the new untracked test file, and execute that instead of the normal one while bisecting.
Hackbraten 4y ago
Have you tried duplicating the test temporarily, which should turn the duplicate into an untracked file so you wouldn’t have to copy it at each step?
32bitkid 4y ago
For the latter I use often use `git stash` for that; stashing off the test case/data, then at each step of the bisect, you can do `git stash apply` (rather than `git stash pop`) to reapply the same changes multiple times. Before moving on, a quick `git reset —-hard` will get back to a clean flip
progers7 4y ago
I have saved a little script to use with git bisect and a stashed test: https://gist.github.com/progers/998452a15180ae369fee338bb39363fd https://gist.github.com/progers/998452a15180ae369fee338bb393...
Deradon 4y ago
For ruby projects where I could simply put the "new test" in a new file I do this: 1. Create new test (e.g. "spec/reproduce-bug_spec.rb") 2. Ignore it locally: `echo "spec/reproduce_bug_spec.rb" >> .git/info/exclude` 3. Run bisect (something like: `git bisect run rspec spec/reproduce-bug_spec.rb`) If running the test gets more complex (e.g. installing dependencies as they might change travelling through history), I usually create a wrapper script (and ignore it) to bisect-run-it.
kazinator 4y ago
To bisect with a new test, you have to do this: $ git bisect [good|bad] $ git cherry-pick new-test $ ./run-the-test $ git reset --hard HEAD^ $ git bisect [good|bad] I sometimes use the stash for that: cherry pick the test as a working tree uncommitted change. "git stash" it before every bisect step, then "git pop".
- alkonaut 4y ago
  Yeah, since it's more or less a standard cumbersome recipe, it seems it would be a good idea to create a high level git switch for it. git bisect run --cherry-pick abc123 [....] to do a regular bisect but at each step cherry picking the commit abc123

k_ 4y ago

This might be a good argument for keeping tests as a separate repository. Which would bring a lot more problems of its own, though...

OJFord 4y ago

Just put the test in a new untracked file?

alkonaut 4y ago
That works but only for languages/build-systems where the mere presence of the test is enough to make it run. In others you might have xml project files, test lists etc which makes the addition of a test require a modification of existing files.
- Izkata 4y ago
  The test subcommand of bisect can do anything you want, so it's easy to make a wrapper that does that prep work before starting that actual tests. Just have to remember to pass back the tests' exit code after cleaning up the repository for git.
- OJFord 4y ago
  Good point, though I don't like the idea of a setup where your test file's only executed if you remember to do that personally (if the tooling doesn't already do similar, I'd at least want to generate the list you describe by globbing the relevantly named/located files). You could just copy that entrypoint into another untracked file, have its list include (or be only - quicker) the new untracked test file, and execute that instead of the normal one while bisecting.

Hackbraten 4y ago

Have you tried duplicating the test temporarily, which should turn the duplicate into an untracked file so you wouldn’t have to copy it at each step?

32bitkid 4y ago

For the latter I use often use `git stash` for that; stashing off the test case/data, then at each step of the bisect, you can do `git stash apply` (rather than `git stash pop`) to reapply the same changes multiple times. Before moving on, a quick `git reset —-hard` will get back to a clean flip

progers7 4y ago

I have saved a little script to use with git bisect and a stashed test: https://gist.github.com/progers/998452a15180ae369fee338bb39363fd https://gist.github.com/progers/998452a15180ae369fee338bb393...

Deradon 4y ago

For ruby projects where I could simply put the "new test" in a new file I do this: 1. Create new test (e.g. "spec/reproduce-bug_spec.rb") 2. Ignore it locally: `echo "spec/reproduce_bug_spec.rb" >> .git/info/exclude` 3. Run bisect (something like: `git bisect run rspec spec/reproduce-bug_spec.rb`) If running the test gets more complex (e.g. installing dependencies as they might change travelling through history), I usually create a wrapper script (and ignore it) to bisect-run-it.

kazinator 4y ago

To bisect with a new test, you have to do this: $ git bisect [good|bad] $ git cherry-pick new-test $ ./run-the-test $ git reset --hard HEAD^ $ git bisect [good|bad] I sometimes use the stash for that: cherry pick the test as a working tree uncommitted change. "git stash" it before every bisect step, then "git pop".

alkonaut 4y ago
Yeah, since it's more or less a standard cumbersome recipe, it seems it would be a good idea to create a high level git switch for it. git bisect run --cherry-pick abc123 [....] to do a regular bisect but at each step cherry picking the commit abc123