3 ms·
well it depends on your error bars. On many modern system you can have +/- 5% variation or more run to run just due to the non-determinism present in modern CP
by deater 17d ago
well it depends on your error bars. On many modern system you can have +/- 5% variation or more run to run just due to the non-determinism present in modern CPU architectures and operating systems (even things like room temperature, time of day, the number of environment variables, etc, can affect this). While you could maybe run a set of careful experiments to characterize and remove this, in my experience most researchers don't bother. So something as small as 3% would need a lot of convincing to me to make the argument that it is significant.
- wat10000 17d agoMultiple separate questions here. First, is a measured improvement actually real or just an artifact of noise? Second, if it is real, is 3% anywhere close to the true value? Third, if 3% is real, is it an important difference? I'm just commenting on the third one. If 3% is real, it's important. I'm inclined to believe there's a real improvement. They made a lot of different measurements. If the measured improvement was a result of noise, you'd expect a lot of variation an a lot of measurements where TSA was actually faster, and then 3% was the average of that variation. There was a lot of variation (expected, because they were measuring different things) but nearly all of them had TSO being either neutral or slower. Looking at their benchmark graphs, I see two (out of dozens) where TSO was faster. As far as being close to the true value, these results suggest there is no single true value, as it depends on the workload. No surprise there.