4 ms·

> Had we moved between -stable branches, bisecting 3+ years of changes could have taken weeks Would it really? Going by their number of 4 hours per bisect step

by jsnell 2y ago

> Had we moved between -stable branches, bisecting 3+ years of changes could have taken weeks Would it really? Going by their number of 4 hours per bisect step, you get 6 bisections per day, which cuts the range to 1/64th of the original. The difference between "three years of changes" and "three weeks of changes" is a factor of 50x. I.e. within one day, they'd already have identified the problematic three week range. After that, the remaining bisection takes exactly as long as this bisection did. Even if they're limited to doing the bisections just doing the working hours in one timezone for some reason, you'd still get those six bisection steps done in just three days. It still would not add weeks.

iainmerrick 2y ago
That's a very good point, but merging in three years of changes would also have pulled in a lot of minor performance changes, and possibly some incompatible changes that would require some code updates. That would slow down each bisection step, and also make it harder to pinpoint the problem. If you know that some small set of recent changes caused a 7% regression, you can be fairly confident there's a single cause. If 3 years of updates cause a 6% or 8% regression (say), it's not obvious that there's going to be a single cause. Even if you find a commit that looks bad, it might already have been addressed in a later commit. Edit to clarify: you're technically correct (the best kind of correct!) but I'd still much prefer to merge 3 weeks rather than 3 years, even though their justification isn't quite right.
wccrawford 2y ago
I take your point, but that assumes someone working 24 hour days, or constantly handing off the project to at least 2 other people every day. I don't think those are reasonable work scenarios, so it's more like 2 bisects (maybe 3!) per day, rather than 6.
- jsnell 2y ago
  Not really, it just assumes that the bisection process is automated. But also, I addressed exactly this objection in the second paragraph :)
  - yencabulator 2y ago
    Bisection and testing might be automated, but resolving merge conflicts isn't.
  - jonhohle 2y ago
    If a bisection takes a day, it would probably take longer to automate than just find it manually. For performance bugs, you may need to look at non-standard metrics or profiling that would otherwise be a one-off and don’t necessarily make sense to automate.
    - jsnell 2y ago
      The full bisection taking just a day doesn't seem compatible with the parameters of the story. Three weeks of FreeBSD changes seems to be about 500 commits. That's about 9 bisection steps. At two steps / day (the best you can do without automation), that's a full work week. It seems obvious that this is worth automating.
drewg123 2y ago
Author here: Note that the 4 hours per bisection is the time to ramp the server up and down and re-image it. It does not count the time to actually do the bisection step. That's because in our case of following head, the merging & conflicts were trivial for each step of the 3 week bisection. Eg, the bisections we're doing are far simpler than the bisections we'd be doing if we didn't track the main branch, and had a far larger patchset that had to be re-merged at each bisection point. I'm cringing just imagining the conflicts.
- crote 2y ago
  Does "the server" imply you're only using a single server for this? I would have expected that at Netflix's scale it wouldn't be too difficult to do the whole bisect-and-test on a couple dozen servers in parallel.
  - generalizations 2y ago
    Wouldn't each bisect depend on the ones before it? So you can't ramp up the next test before you finish the prior.
    - crote 2y ago
      Oh right, that's obviously not going to work unless you're willing to waste a huge amount of resources on speculative execution. The fact that the selection of the next bisect depends on the result of the current one completely slipped my mind!
    - toast0 2y ago
      You could speculate in both directions, and test three revisions in parallel... However, if there's a lot of manual merging to do, you might not want to do the extra merge that's involved there. You might also get nerd sniped into figuring out if there's a better pattern if you're going to test multiple revisions in parallel.
zellyn 2y ago
lol, came here to say this, armed with identical log2 53 == 5.7 :-) The replies to your comment are of course, spot on, though. Finding 8% of performance regression in three years of commits could have taken a looooong time.

by jsnell 2y ago

iainmerrick 2y ago

That's a very good point, but merging in three years of changes would also have pulled in a lot of minor performance changes, and possibly some incompatible changes that would require some code updates. That would slow down each bisection step, and also make it harder to pinpoint the problem. If you know that some small set of recent changes caused a 7% regression, you can be fairly confident there's a single cause. If 3 years of updates cause a 6% or 8% regression (say), it's not obvious that there's going to be a single cause. Even if you find a commit that looks bad, it might already have been addressed in a later commit. Edit to clarify: you're technically correct (the best kind of correct!) but I'd still much prefer to merge 3 weeks rather than 3 years, even though their justification isn't quite right.

wccrawford 2y ago

I take your point, but that assumes someone working 24 hour days, or constantly handing off the project to at least 2 other people every day. I don't think those are reasonable work scenarios, so it's more like 2 bisects (maybe 3!) per day, rather than 6.

jsnell 2y ago
Not really, it just assumes that the bisection process is automated. But also, I addressed exactly this objection in the second paragraph :)
- yencabulator 2y ago
  Bisection and testing might be automated, but resolving merge conflicts isn't.
- jonhohle 2y ago
  If a bisection takes a day, it would probably take longer to automate than just find it manually. For performance bugs, you may need to look at non-standard metrics or profiling that would otherwise be a one-off and don’t necessarily make sense to automate.
  - jsnell 2y ago
    The full bisection taking just a day doesn't seem compatible with the parameters of the story. Three weeks of FreeBSD changes seems to be about 500 commits. That's about 9 bisection steps. At two steps / day (the best you can do without automation), that's a full work week. It seems obvious that this is worth automating.

drewg123 2y ago

Author here: Note that the 4 hours per bisection is the time to ramp the server up and down and re-image it. It does not count the time to actually do the bisection step. That's because in our case of following head, the merging & conflicts were trivial for each step of the 3 week bisection. Eg, the bisections we're doing are far simpler than the bisections we'd be doing if we didn't track the main branch, and had a far larger patchset that had to be re-merged at each bisection point. I'm cringing just imagining the conflicts.

crote 2y ago
Does "the server" imply you're only using a single server for this? I would have expected that at Netflix's scale it wouldn't be too difficult to do the whole bisect-and-test on a couple dozen servers in parallel.
- generalizations 2y ago
  Wouldn't each bisect depend on the ones before it? So you can't ramp up the next test before you finish the prior.
  - crote 2y ago
    Oh right, that's obviously not going to work unless you're willing to waste a huge amount of resources on speculative execution. The fact that the selection of the next bisect depends on the result of the current one completely slipped my mind!
  - toast0 2y ago
    You could speculate in both directions, and test three revisions in parallel... However, if there's a lot of manual merging to do, you might not want to do the extra merge that's involved there. You might also get nerd sniped into figuring out if there's a better pattern if you're going to test multiple revisions in parallel.

zellyn 2y ago

lol, came here to say this, armed with identical log2 53 == 5.7 :-) The replies to your comment are of course, spot on, though. Finding 8% of performance regression in three years of commits could have taken a looooong time.