5 ms·
> You're not being quite honest here: of course just knowing if something compiles at all, whilst nice, is not that massive a difference (because you can fairly
by lmm 6y ago
> You're not being quite honest here: of course just knowing if something compiles at all, whilst nice, is not that massive a difference (because you can fairly easily script an automated bisect that will skip non-compiling commits by itself). But that's much weaker than what I actually said: knowing that the commit has passed your (1.5 h) CI, and making it so that all your merge commits on master will have this property.
In the (somewhat exceptional) case I'm talking about it was 1.5h for a standard build; IIRC straight compilation was more than half of it. In any case, there's no real difference between compilation and "CI" here; you make your bisect script run whatever your CI test is, and skip if it fails, before running the part you're actually testing.
> Since you apparently haven't tried it, maybe you should be a bit more careful about dismissing the utility out of hand?
I've tried workflows that made it easy to do a bisect that tests only commits from the history of master; unless you can explain how what you're suggesting achieves something better than that, I don't think not having used your precise script invalidates my views.
In fact it seems to me that your workflow is distinctly worsened by using rebase; if you used merges then some intermediate commits from feature branches would have been successfully built by CI (pre-merge builds, builds of "early review" PRs that were reworked before merging to master, builds that the developer deliberately ran on CI for whatever reason), whereas by using rebase you guarantee that only the post-merge states of master are available to you for bisection.
> if you hit a commit that's broken in a more subtle fashion than "does not compile" but that CI would have found it's not always cheap to work out if you hit the regression or some unrelated temporary breakage.
What are you doing on CI that's so different to what you're doing during local development? If breakages don't show up until you make the PR that you want to merge that's bad for everyone; running the unit tests that pertain to the code you're working on, if not the whole suite, before committing is just common sense. Of course it's possible for something to work locally and break on CI, but that's a very rare case (much rarer than generally-flaky tests, IME).
> You are also making a several implicit assumptions which are unlikely to hold: it is not true that typically none of the utility of bisect manifests before you have identified the precise lines of code that caused an issue and understood them. If you have some acute problem in production, being able to reliably and quickly locate the feature branch that introduced it is very valuable (for getting the right people to look at the problem or even shipping some work around before the problem is fully diagnosed). Often certainly a recent deployment is to blame. But sometimes a deployment can exercise something that regressed much earlier and trigger a data corruption that is not noticed immediately.
Narrowing it down to a branch really isn't much quicker than narrowing it down to a commit - you've already come up with the test case/script, so it's just a case of letting it run for maybe 5 more cases (if we assume maybe 30 commits on the feature branch). If you have a very small team then I guess narrowing it down to a specific branch might be much quicker than narrowing it down to a commit within that branch - but in that case the bisect is going to find the same thing, you can see when it's got to the stage of testing commits from the same branch and start investigating there. And if you really want to bisect just via the history of master, you can always do that (admittedly with a little scripting, but you don't seem to be shy of that).
> Also, of course once you found some problematic merge there is no reason you wouldn't then continue to use bisect to find the precise commit in the feature branch!
But you can't do that if you've rebased the branch, because most of the branch history is (often) broken. If you were squash-merging you could dig out the "original" version of the branch (assuming it's not been gced) and bisect there (assuming the problem is solely due to a change on that branch and not an interaction between that branch and a concurrent change on master), but if you're rebasing you can't even do that, because if developers are in the habit of rebasing then the "original" branch was probably rebased and force-pushed as well, so is likely to have old commits that don't compile.
> Assuming each developer lands something on master every two days on average, you'd have to deal with 250 merges a day. Say you can run about 5 sequential CI runs per work day, you'd need to test pretty large batches of ~50PRs per batch, which is more than I have experience with. Since you will only batch stuff that passed branch CI, and disregarding flakes for the moment, a batch should only fail because of an incompatibility of PRs within the same batch or against some very recent addition to master. So you can probably get the failure rate into the low single digit percentages, at which you'd need to do run enough alternative batches in parallel to cope with one or two bad PRs per batch. That still seems feasible if you put batches together intelligently
Yeah, that kind of approach definitely seems viable - we had a design like that sketched out, and may even have prototyped it at one point. It was decided that it wasn't worthwhile, because we would still have needed our process for catching when flaky tests were introduced, and that process caught conflicts like this "for free". So you are probably right that it is achievable to avoid ever merging non-compiling code to master except when meta problems happen, but I don't think that actually takes you significantly closer to "master is never broken".
> Sure. But minimizing those bad commits (and probably even marking them after the discovery, e.g. via git-notes) pays off.
Yes and no; as long as they're isolated, I don't think pushing the rate of bad commits from 1% down to 0.5% or even 0.1% is really a game-changer. Obviously there's a threshold where you have enough bad commits that it significantly disrupts being able to bisect at all, but unless you can manage to completely eliminate the possibility of a broken commit, there's a wide spread of bad-commit rates where your workflow has to be pretty much the same. Just like how buying higher quality hardware to reduce defect rates is generally not worthwhile unless you go all the way to buying a mainframe with fully redundant everything and guaranteed uptime; you still need to be fault-tolerant if you buy the pro component with 99% uptime, so you might as well buy the consumer version with 98% uptime if it's even 5% cheaper.
- patrec 6y ago> Yes and no; as long as they're isolated, I don't think pushing the rate of bad commits from 1% down to 0.5% or even 0.1% is really a game-changer. But how are they going to be isolated if your CI runs for 1.5h and you allow bad commits to master? Unless your CI magically stops very early on a logical merge conflict that does not manifest as patch-level merge conflict, you will make bad merges to master for the 1.5h it will take you to figure out master is broken. Based on the above numbers (250 merges a day) that would be 50 bad merges, assuming ~4 commits per merge you get 1000 broken commits on master on a single broken build; and I can't see how this is not going to hurt your ability to bisect in practice. There are plenty of nice things that become practical and easy if you combine very low rate of bad merges and if you can get previously compiled artifacts for every commits in seconds to sub-seconds (easy to do with nix) and have an optimized spin-up. E.g. you can very quickly find performance regressions via bisection that happened months ago (useful if you have a lot of seasonal variation in performance profiles (as in search or lots of retail) a perf regression might go unnoticed for considerable time). > Narrowing it down to a branch really isn't much quicker than narrowing it down to a commit - you've already come up with the test case/script, so it's just a case of letting it run for maybe 5 more cases I'm failing to get my point across. It's not just 5 more cases, one is reliable the other isn't. I can find the branch completely reliably. And even if I have to deal with potentially broken commits in the branch at that point, this is much less hassle then doing so before I have found the right branch, because a) they are going to be far fewer, so I have a lower chance of hitting one than when bisecting of 10k commits or whatever b) I have some meaningful context. I know I'm looking at feature X and I can probably pretty quickly figure out if some failure inside is the thing I'm looking for or some spurious bad commit. I.e. once I've narrowed it down it's much cheaper to figure out then if I stopped somewhere random in those 10k commits. > But you can't do that if you've rebased the branch, because most of the branch history is (often) broken. Well, except it isn't :) Why would it be? Since I avoid long lived branches and tend to coordinate with co-workers when modifying a particular part of the code base, it's not like I would have to deal with merge conflicts all the time, and even in cases where I am, I'd generally try to fix the right commit in the rebase, rather than introducing a chain of broken commits. Seriously, there are plenty of reasons commits on a feature branch will be broken some of the time (e.g. you do a larger change that involves some intermediate step were things are broken) and I don't think rebase materially affects the total adversely in my typical workflows (it certainly also reduces bad commits, because if I notice something is broken on my branch I'll try to fix the relevant commit, which would not be allowed in your workflow). > What are you doing on CI that's so different to what you're doing during local development? I try to maximize my development speed with local testing. If I can catch 90% of errors for 5% of test time or less, I'll do that ever time and rely on CI to inform me about more subtle errors (e.g. either integration test failures or higher numbers of samples for property based tests). Why would I not farm out long-running tests to a dedicated CI box and continue working in the meantime?