4 ms·
You make a good point I hadn't considered. In that case, not even robust least squares will save you, because the samples are not independent of one another. Yo
by cscheid 7y ago
You make a good point I hadn't considered. In that case, not even robust least squares will save you, because the samples are not independent of one another. You'd have to start looking into explicit time-dependent methods and yikes, I have no idea of what the literature says about time dependence and outliers. I wouldn't be surprised if it's "here be dragons" territory.
- colmmacc 7y agoSpeculative execution optimizations, e.g. branch prediction, are even data dependent. It's a very complex measurement space!
- gdxhyrd 7y agoYou are speaking as a stats person. :) While there are dozens of things that can go south in benchmarking, how to solve them is not about applying advanced stats, but about understanding and eliminating the sources of noise. Most benchmarks in CS can be done quite easily as long as one understands all the technology stack.
- cscheid 7y agoI'm not a stats person! But I have been burned in real projects by doing the wrong thing. I just have learned --- the hard way --- to honestly appreciate it.