6 ms·
> A stochastic search process with an executable optimization objective over space of programs S can only maintain or improve the objective Reasons this doesn'
by hiddencost 1mo ago
> A stochastic search process with an executable optimization objective over space of programs S can only maintain or improve the objective
Reasons this doesn't follow:
(1) Benchmarks never match real world use, and many optimizations the improve benchmarks degrade cases that aren't measured (think about how CPU cache behavior can be surprising)
(2) In software performance optimization, frequently there is significant noise, from many sources. This makes it difficult to guarantee that a measured change is actually an improvement.
- fc417fc802 1mo agoBoth of those things are indicators of deficiencies in the testing process.