4 ms·

> The slowest and the fastest runs are discarded. The mean value of the remaining three runs is the result for that contender and will be added to the leaderboa

> The slowest and the fastest runs are discarded. The mean value of the remaining three runs is the result for that contender and will be added to the leaderboard. I think it's better to discard the two slowest, or simply accept the fastest as the correct. There's (in my opinion) no good reason to discard the best runs.

mayank 3y ago
This is a pretty standard measure called the Trimmed Mean: https://statisticsbyjim.com/basics/trimmed-mean/ https://statisticsbyjim.com/basics/trimmed-mean/
- stabbles 3y ago
  For performance benchmarking the minimal runtime is typically the best estimator if the computations are identical, cause it measures perf w/o interrupts. If the language is garbage collected, or if the test is randomized you obviously don't want to look at the minimum.
  - mayank 3y ago
    > the minimal runtime is typically the best estimator Depends what you’re estimating. The minimum is usually not representative of “real world” performance, which is why we use measures of central tendency over many runs for performance benchmarks.
- ProblemFactory 3y ago
  Variability in software runtime arises mostly from other software running on the same system. If you are looking for a real-world, whole-system benchmark (like a database or app server), then taking the average makes sense. If you are benchmarking an individual algorithm or program and its optimisations, then taking the fastest run makes sense - that was the run with least external interference. The only exception might be if you want to benchmark with cold caches, but then you need to reset these carefully between runs as well.
hyperpape 3y ago
There are good reasons to discard the best runs. If you think of your system as having predictable behavior that's slowed down by background processing happening on the machine, it might make sense to use the best run. But if there are any intrinsic sources of non-determinism in the program (and it's far more likely than you might think), taking the best time is likely to be unrepresentative. https://tratt.net/laurie/blog/2019/minimum_times_tend_to_mislead_when_benchmarking.html https://tratt.net/laurie/blog/2019/minimum_times_tend_to_mis... is a good piece on the subject.
paxys 3y ago
If you don't think discarding the fastest is acceptable then why are you in favor of discarding the slowest?
- JohnKemeny 3y ago
  Because there can be outliers in terms of slowness; perhaps the CPU gets a hickup, the GC runs an abnormal amount of extra times, the kernel decides to reschedule stuff, whatever, I don't know much about these things. There can't be outliers in terms of fastness; the CPU doesn't accidentally run the program much faster. But then again, what the hell do I know...
  - deleted 3y ago
    [deleted]
  - paxys 3y ago
    It's the same logic both ways. The OS runs 10 background tasks on average at any given time. On one of your runs it was doing 15 things, while on another it was doing 5 things. They are both outliers.

mayank 3y ago

This is a pretty standard measure called the Trimmed Mean: https://statisticsbyjim.com/basics/trimmed-mean/ https://statisticsbyjim.com/basics/trimmed-mean/

stabbles 3y ago
For performance benchmarking the minimal runtime is typically the best estimator if the computations are identical, cause it measures perf w/o interrupts. If the language is garbage collected, or if the test is randomized you obviously don't want to look at the minimum.
- mayank 3y ago
  > the minimal runtime is typically the best estimator Depends what you’re estimating. The minimum is usually not representative of “real world” performance, which is why we use measures of central tendency over many runs for performance benchmarks.
ProblemFactory 3y ago
Variability in software runtime arises mostly from other software running on the same system. If you are looking for a real-world, whole-system benchmark (like a database or app server), then taking the average makes sense. If you are benchmarking an individual algorithm or program and its optimisations, then taking the fastest run makes sense - that was the run with least external interference. The only exception might be if you want to benchmark with cold caches, but then you need to reset these carefully between runs as well.

hyperpape 3y ago

There are good reasons to discard the best runs. If you think of your system as having predictable behavior that's slowed down by background processing happening on the machine, it might make sense to use the best run. But if there are any intrinsic sources of non-determinism in the program (and it's far more likely than you might think), taking the best time is likely to be unrepresentative. https://tratt.net/laurie/blog/2019/minimum_times_tend_to_mislead_when_benchmarking.html https://tratt.net/laurie/blog/2019/minimum_times_tend_to_mis... is a good piece on the subject.

paxys 3y ago

If you don't think discarding the fastest is acceptable then why are you in favor of discarding the slowest?

JohnKemeny 3y ago
Because there can be outliers in terms of slowness; perhaps the CPU gets a hickup, the GC runs an abnormal amount of extra times, the kernel decides to reschedule stuff, whatever, I don't know much about these things. There can't be outliers in terms of fastness; the CPU doesn't accidentally run the program much faster. But then again, what the hell do I know...
- deleted 3y ago
  [deleted]
- paxys 3y ago
  It's the same logic both ways. The OS runs 10 background tasks on average at any given time. On one of your runs it was doing 15 things, while on another it was doing 5 things. They are both outliers.