21 ms·
> Yes and no; as long as they're isolated, I don't think pushing the rate of bad commits from 1% down to 0.5% or even 0.1% is really a game-changer. But how ar
by patrec 6y ago
> Yes and no; as long as they're isolated, I don't think pushing the rate of bad commits from 1% down to 0.5% or even 0.1% is really a game-changer.
But how are they going to be isolated if your CI runs for 1.5h and you allow bad commits to master? Unless your CI magically stops very early on a logical merge conflict that does not manifest as patch-level merge conflict, you will make bad merges to master for the 1.5h it will take you to figure out master is broken. Based on the above numbers (250 merges a day) that would be 50 bad merges, assuming ~4 commits per merge you get 1000 broken commits on master on a single broken build; and I can't see how this is not going to hurt your ability to bisect in practice.
There are plenty of nice things that become practical and easy if you combine very low rate of bad merges and if you can get previously compiled artifacts for every commits in seconds to sub-seconds (easy to do with nix) and have an optimized spin-up. E.g. you can very quickly find performance regressions via bisection that happened months ago (useful if you have a lot of seasonal variation in performance profiles (as in search or lots of retail) a perf regression might go unnoticed for considerable time).
> Narrowing it down to a branch really isn't much quicker than narrowing it down to a commit - you've already come up with the test case/script, so it's just a case of letting it run for maybe 5 more cases
I'm failing to get my point across. It's not just 5 more cases, one is reliable the other isn't. I can find the branch completely reliably. And even if I have to deal with potentially broken commits in the branch at that point, this is much less hassle then doing so before I have found the right branch, because a) they are going to be far fewer, so I have a lower chance of hitting one than when bisecting of 10k commits or whatever b) I have some meaningful context. I know I'm looking at feature X and I can probably pretty quickly figure out if some failure inside is the thing I'm looking for or some spurious bad commit. I.e. once I've narrowed it down it's much cheaper to figure out then if I stopped somewhere random in those 10k commits.
> But you can't do that if you've rebased the branch, because most of the branch history is (often) broken.
Well, except it isn't :) Why would it be? Since I avoid long lived branches and tend to coordinate with co-workers when modifying a particular part of the code base, it's not like I would have to deal with merge conflicts all the time, and even in cases where I am, I'd generally try to fix the right commit in the rebase, rather than introducing a chain of broken commits. Seriously, there are plenty of reasons commits on a feature branch will be broken some of the time (e.g. you do a larger change that involves some intermediate step were things are broken) and I don't think rebase materially affects the total adversely in my typical workflows (it certainly also reduces bad commits, because if I notice something is broken on my branch I'll try to fix the relevant commit, which would not be allowed in your workflow).
> What are you doing on CI that's so different to what you're doing during local development?
I try to maximize my development speed with local testing. If I can catch 90% of errors for 5% of test time or less, I'll do that ever time and rely on CI to inform me about more subtle errors (e.g. either integration test failures or higher numbers of samples for property based tests). Why would I not farm out long-running tests to a dedicated CI box and continue working in the meantime?
- lmm 6y ago> assuming ~4 commits per merge you get 1000 broken commits on master on a single broken build No, because none of the commits from the feature branches are broken - only the merge commits themselves are broken. Since a merge commit by definition has multiple parents i.e. multiple paths through it, they're the least damaging place for breaks to occur; when you hit a broken merge commit it's usually easy for the bisect to find a different path around. > I'm failing to get my point across. It's not just 5 more cases, one is reliable the other isn't. I can find the branch completely reliably. And even if I have to deal with potentially broken commits in the branch at that point, this is much less hassle then doing so before I have found the right branch, because a) they are going to be far fewer, so I have a lower chance of hitting one than when bisecting of 10k commits or whatever b) I have some meaningful context. I know I'm looking at feature X and I can probably pretty quickly figure out if some failure inside is the thing I'm looking for or some spurious bad commit. I.e. once I've narrowed it down it's much cheaper to figure out then if I stopped somewhere random in those 10k commits. Narrowing it down within the branch by thinking through it in your head is rarely going to be quicker than just letting the automated bisect run for a few more cases, and never going to be less effort. At that stage you've already done all the work. Let the computer do what it's good at. > Well, except it isn't :) Why would it be? Since I avoid long lived branches and tend to coordinate with co-workers when modifying a particular part of the code base, it's not like I would have to deal with merge conflicts all the time, and even in cases where I am, I'd generally try to fix the right commit in the rebase, rather than introducing a chain of broken commits. If you get a git-level conflict, sure. The bigger problems are the semantic conflicts that don't show up as git conflicts e.g. something that was used in your changes gets renamed - I've never seen anyone who goes back and fixes that properly after a rebase. In a large codebase with a lot of developers who believe in aggressive refactoring (which I think is 100% worthwhile, though I suppose that's a separate discussion) this will happen a lot. > I try to maximize my development speed with local testing. If I can catch 90% of errors for 5% of test time or less, I'll do that ever time and rely on CI to inform me about more subtle errors (e.g. either integration test failures or higher numbers of samples for property based tests). Why would I not farm out long-running tests to a dedicated CI box and continue working in the meantime? Longer-running regression tests, sure, but those tests don't block bisection either. To me "good enough to continue doing development work on" and "good enough to bisect with" are the same standard; if it's completely broken (e.g. non-compiling) then I can't get anywhere, if there's a slight flakiness that shows up in some occasional edge cases then it doesn't really matter. I don't understand being happy to continue work during regular development, but throwing out anything that's not 1000% reliable when you're bisecting.