3 ms·
A couple of minor typos and nits: The for loop body should use > $i Inconsistent use of redirection (which requires a root shell) and sudo <command> | tee, wh
by sfink 7y ago
A couple of minor typos and nits:
The for loop body should use > $i
Inconsistent use of redirection (which requires a root shell) and sudo <command> | tee, which doesn't.
I think the S in SMT is for symmetric
And a comment - I liked the mention of mean vs minimum, though I would have liked the median to be included as well, and more info on when and why each may be more appropriate. Minimum is good when you want to answer "how much time is required to do just what is required for what this code is trying to accomplish". The minimum is the closest approximation you can get; the "true time required" can never be more than the minimum. Mean is good for something you'll be doing a lot of, and want to predict how much total time it'll take. Median gives you a realistic measure of how long a task will take; if you tell someone that phase X will take 3 seconds, and 3 is the median, then you'll be too high half the time and too low the other half, which is better than reporting the minimum which will be too low nearly 100% of the time. But then again, your article is more about eliminating noise when measuring the effects of changes, so minimum is probably generally the most reasonable thing to be looking at. (Though if it's a large phase with many constituent sources of variance, you have to be careful because your minimum will keep shrinking the more repetitions you do!)
- rrss 7y ago> I think the S in SMT is for symmetric It's definitely for simultaneous. You are probably thinking of SMP (symmetric multiprocessing).
- pstch 7y agoMinor nitpick: I think you meant to write "<command> | sudo tee". And a question on the comment - In general (I don't have a lot of benchmarking experience, I did a few of them and used lower quartile to reduce the impact from interference), is it hard to determine that the minimum is not shrinking anymore ? I suppose we could run a benchmark until we have a certain amount (say 3, but it will depend on the distribution) of relatively close minimum values ? This seems to be a good way to have a good probability of obtaining the "true time required" you mentioned.
- mh- 7y ago>The minimum is the closest approximation you can get; the "true time required" can never be more than the minimum. I think you meant to say "never be less than the minimum." Agree about its utility here.
- lixtra 7y agoIf the performance bounces around, how do you know you hit the true minimum? Therefore the true time can be less than the observed minimum. On the other hand, the true time required fits in the observed minimum, so it cannot be more.