4 ms·

Biasing a binary search would only be beneficial if you know something about the distribution of the search space

by pacaro 3y ago

bgirard 3y ago
If the factor in one direction is large enough then a linear search becomes more efficient. Say you have 20 commits remaining and the factor is 1,000x more costly to make it easier to picture. You're better off doing a linear search which guarantees you'll spend less than 2,000x searching the space. That suggests that for a larger search space with a large enough difference, the optimal bisection point is probably not always the midpoint even if you know nothing about the distribution. Perhaps someone can find the exact formula for selecting the next revision to search?
- jwilk 3y ago
  > You're better off doing a linear search which guarantees you'll spend less than 2,000x searching the space. Almost. If only the last commit is slow, binary search is still faster.
  - bgirard 3y ago
    > better off Better off as in expected/average case. Good point, but only marginally better in the worse case.
mortehu 3y ago
Each boot updates your empirical distribution. As a trivial example, if you have booted a version 9999 times with no hanging, a later version will likely give you more information per boot.
electroly 3y ago
There's an additional stopping problem here that isn't present in a normal binary search. Binary search assumes you can do a test and know for sure whether you've found the target item, a lower item, or a higher item. If the test itself is stochastic and you don't know how long you have to run it to get the hang, I'd think you'd get results faster by running commits randomly and excluding them from consideration when they hang. Effectively, you're running all the commits at the same time instead of working on one commit and not moving on until you've made a decision on it. Then at any time you will have a list of commits that have hanged and a list of commits that have not hanged yet, and you can keep the entire experiment running arbitarily long to catch the long-tail effects rather than having to choose when to stop testing a single non-hanging commit and move onto the next one.
- pacaro 3y ago
  I can see some interesting approaches here. Given n threads/workers you could divide the search space into n sample points (for simplicity let's divide it evenly) and run the repeated test on each point. When a point hangs, that establishes a new upper limit, all higher search points are eliminated, the workers reassigned in the remaining search space. Given the uncertainty I can see how this might be more efficient, especially if the variance of the heisenbug is high.

Biasing a binary search would only be beneficial if you know something about the distribution of the search space

by pacaro 3y ago

Biasing a binary search would only be beneficial if you know something about the distribution of the search space

If the factor in one direction is large enough then a linear search becomes more efficient. Say you have 20 commits remaining and the factor is 1,000x more costly to make it easier to picture. You're better off doing a linear search which guarantees you'll spend less than 2,000x searching the space. That suggests that for a larger search space with a large enough difference, the optimal bisection point is probably not always the midpoint even if you know nothing about the distribution. Perhaps someone can find the exact formula for selecting the next revision to search?

jwilk 3y ago
> You're better off doing a linear search which guarantees you'll spend less than 2,000x searching the space. Almost. If only the last commit is slow, binary search is still faster.
- bgirard 3y ago
  > better off Better off as in expected/average case. Good point, but only marginally better in the worse case.

mortehu 3y ago

Each boot updates your empirical distribution. As a trivial example, if you have booted a version 9999 times with no hanging, a later version will likely give you more information per boot.

electroly 3y ago

There's an additional stopping problem here that isn't present in a normal binary search. Binary search assumes you can do a test and know for sure whether you've found the target item, a lower item, or a higher item. If the test itself is stochastic and you don't know how long you have to run it to get the hang, I'd think you'd get results faster by running commits randomly and excluding them from consideration when they hang. Effectively, you're running all the commits at the same time instead of working on one commit and not moving on until you've made a decision on it. Then at any time you will have a list of commits that have hanged and a list of commits that have not hanged yet, and you can keep the entire experiment running arbitarily long to catch the long-tail effects rather than having to choose when to stop testing a single non-hanging commit and move onto the next one.

pacaro 3y ago
I can see some interesting approaches here. Given n threads/workers you could divide the search space into n sample points (for simplicity let's divide it evenly) and run the repeated test on each point. When a point hangs, that establishes a new upper limit, all higher search points are eliminated, the workers reassigned in the remaining search space. Given the uncertainty I can see how this might be more efficient, especially if the variance of the heisenbug is high.