3 ms·

I know it's unconventional, but I like to keep my (anything larger than unit) tests in a separate repo so that I'm not running old versions of the tests when I

I know it's unconventional, but I like to keep my (anything larger than unit) tests in a separate repo so that I'm not running old versions of the tests when I bisect: - find bug - write new test to expose it - bisect commits, running new test each time - find culprit commit - stop bisecting, fix bug - when the test passes, you know bug is fixed.

tuetuopay 4y ago
How do you handle tests against new features that would definitely not pass on commits before the feature is introduced?
- matthews2 4y ago
  Some test frameworks allow you to mark these tests as xfail (expected fail). When you fix the issue, you can remove the xfail in the same commit.
  - SloopJon 4y ago
    This was my preferred approach for several years. The tests were in a repo separate from the code under test, and each declared its prerequisites, either as a feature or a version pattern. If a test started failing, I could annotate it as an xfail with an open-ended version pattern. If I noticed that an xfail started passing, making it a upass (I think these were blue in our tests results), I put an upper bound on the xfail version pattern. The nice thing about this was that I could always run the latest and greatest suite against any version, although it made it a little more complicated to run them as they existed when they were first run.
- ygra 4y ago
  I'd typically start bisecting only after (or at the changeset) the feature has been introduced. At least with hg (and seemingly git as well) you can tell bisect to use a commit range where one end is good and the other is bad.
- __MatrixMan__ 4y ago
  If I'm bisecting, I usually already have a single test in mind which exposes the bug I'm looking for, so I'm explicitly running that test only. In general, I want to store one or more services and a first/last valid commit for each one as metadata for each test so that my framework knows which ones to run based on the environment under test. Haven't added this feature yet though.
Karellen 4y ago
My preferred variant of this would be - write new tests to expose it, but not commit the tests - bisect commits, running new tests each time - find culprit commit - stop bisecting, fix bug - Look for similar bugs nearby, and ways to improve the new tests, now I know the root problem. - Figure out the best series of commits for the tests and fixes together, that will be easiest to review by my colleagues/future me/alien archaeologists. - Send PRs/commit to main.

tuetuopay 4y ago

How do you handle tests against new features that would definitely not pass on commits before the feature is introduced?

matthews2 4y ago
Some test frameworks allow you to mark these tests as xfail (expected fail). When you fix the issue, you can remove the xfail in the same commit.
- SloopJon 4y ago
  This was my preferred approach for several years. The tests were in a repo separate from the code under test, and each declared its prerequisites, either as a feature or a version pattern. If a test started failing, I could annotate it as an xfail with an open-ended version pattern. If I noticed that an xfail started passing, making it a upass (I think these were blue in our tests results), I put an upper bound on the xfail version pattern. The nice thing about this was that I could always run the latest and greatest suite against any version, although it made it a little more complicated to run them as they existed when they were first run.
ygra 4y ago
I'd typically start bisecting only after (or at the changeset) the feature has been introduced. At least with hg (and seemingly git as well) you can tell bisect to use a commit range where one end is good and the other is bad.
__MatrixMan__ 4y ago
If I'm bisecting, I usually already have a single test in mind which exposes the bug I'm looking for, so I'm explicitly running that test only. In general, I want to store one or more services and a first/last valid commit for each one as metadata for each test so that my framework knows which ones to run based on the environment under test. Haven't added this feature yet though.

Karellen 4y ago

My preferred variant of this would be - write new tests to expose it, but not commit the tests - bisect commits, running new tests each time - find culprit commit - stop bisecting, fix bug - Look for similar bugs nearby, and ways to improve the new tests, now I know the root problem. - Figure out the best series of commits for the tests and fixes together, that will be easiest to review by my colleagues/future me/alien archaeologists. - Send PRs/commit to main.