4 ms·

"zero-tolerance performance regression policy"... no patch can land if it regresses benchmarked performance. I'm guessing the tooling around this used subversi

by tln 4y ago

"zero-tolerance performance regression policy"... no patch can land if it regresses benchmarked performance. I'm guessing the tooling around this used subversion's increasing commit numbers and it was easier to add a shim to git, than to rewrite or rethink the tooling.

usefulcat 4y ago
> no patch can land if it regresses benchmarked performance ..unless it fixes some important security vulnerability, one hopes..
- mhh__ 4y ago
  the trick is to change the benchmark at the same time
- btown 4y ago
  Via https://webkit.org/performance/ https://webkit.org/performance/ : > If a patch lands that regresses performance according to our benchmarks, then the person responsible must either back the patch out of the tree or drop everything immediately and fix the regression. I imagine that a security hotfix would lead almost immediately to the second situation (perhaps as soon as the implementor had gotten some sleep!)
xfmpXIe76lF4GfR 4y ago
git literally has built-in tooling for this. It's called bisect (and they literally mention "bisection" in the next sentence).
- howinteresting 4y ago
  The builtin tooling is insufficient for many purposes, including if your bisect algorithm requires you to run tests across many machines. Many large projects write their own bisect framework because of this.
  - Too 4y ago
    It is intuitively a bit hard to believe, but bisecting in parallel is actually not much faster than serial. In my experience you just save the final one or two steps in the bisection - regardless of how long the range is! Imagine bisecting builds with parallelism of two, when one of them completes, there is a 50% chance that the build on the other side of the range is now uninteresting. If you are lucky and it is interesting, you’ve only saved yourself half a step in the bisection, because when running in parallel you sliced the range in 3 rather than 2. Adding even more parallelism just makes this effect even worse. Someone can probably work out the math better than me but you can quickly see that for 2x build power you instantly waste half the results for very marginal gain. Just comparing the big O should also tell you this, parallelism only buys you O(N) while bisecting is O(log N).
gernb 4y ago
> no patch can land if it regresses benchmarked performance. That's complete BS. Just go search for all the perf regressions in the issue tracker.
- kelnos 4y ago
  > benchmarked Perhaps the regressions in the issue tracker are for things without benchmarks.
alerighi 4y ago
A security fix most of the times does have a performance impact. What do they do?

by tln 4y ago

usefulcat 4y ago

> no patch can land if it regresses benchmarked performance ..unless it fixes some important security vulnerability, one hopes..

mhh__ 4y ago
the trick is to change the benchmark at the same time
btown 4y ago
Via https://webkit.org/performance/ https://webkit.org/performance/ : > If a patch lands that regresses performance according to our benchmarks, then the person responsible must either back the patch out of the tree or drop everything immediately and fix the regression. I imagine that a security hotfix would lead almost immediately to the second situation (perhaps as soon as the implementor had gotten some sleep!)

xfmpXIe76lF4GfR 4y ago

git literally has built-in tooling for this. It's called bisect (and they literally mention "bisection" in the next sentence).

howinteresting 4y ago
The builtin tooling is insufficient for many purposes, including if your bisect algorithm requires you to run tests across many machines. Many large projects write their own bisect framework because of this.
- Too 4y ago
  It is intuitively a bit hard to believe, but bisecting in parallel is actually not much faster than serial. In my experience you just save the final one or two steps in the bisection - regardless of how long the range is! Imagine bisecting builds with parallelism of two, when one of them completes, there is a 50% chance that the build on the other side of the range is now uninteresting. If you are lucky and it is interesting, you’ve only saved yourself half a step in the bisection, because when running in parallel you sliced the range in 3 rather than 2. Adding even more parallelism just makes this effect even worse. Someone can probably work out the math better than me but you can quickly see that for 2x build power you instantly waste half the results for very marginal gain. Just comparing the big O should also tell you this, parallelism only buys you O(N) while bisecting is O(log N).

gernb 4y ago

> no patch can land if it regresses benchmarked performance. That's complete BS. Just go search for all the perf regressions in the issue tracker.

kelnos 4y ago
> benchmarked Perhaps the regressions in the issue tracker are for things without benchmarks.

alerighi 4y ago

A security fix most of the times does have a performance impact. What do they do?