3 ms·
Is this just a question of what one considers to be “significant”? I’d consider 3% to be significant but the authors apparently don’t.
by wat10000 8d ago
Is this just a question of what one considers to be “significant”? I’d consider 3% to be significant but the authors apparently don’t.
- deater 8d agowell it depends on your error bars. On many modern system you can have +/- 5% variation or more run to run just due to the non-determinism present in modern CPU architectures and operating systems (even things like room temperature, time of day, the number of environment variables, etc, can affect this). While you could maybe run a set of careful experiments to characterize and remove this, in my experience most researchers don't bother. So something as small as 3% would need a lot of convincing to me to make the argument that it is significant.
- wat10000 8d agoMultiple separate questions here. First, is a measured improvement actually real or just an artifact of noise? Second, if it is real, is 3% anywhere close to the true value? Third, if 3% is real, is it an important difference? I'm just commenting on the third one. If 3% is real, it's important. I'm inclined to believe there's a real improvement. They made a lot of different measurements. If the measured improvement was a result of noise, you'd expect a lot of variation an a lot of measurements where TSA was actually faster, and then 3% was the average of that variation. There was a lot of variation (expected, because they were measuring different things) but nearly all of them had TSO being either neutral or slower. Looking at their benchmark graphs, I see two (out of dozens) where TSO was faster. As far as being close to the true value, these results suggest there is no single true value, as it depends on the workload. No surprise there.