3 ms·
This is a pretty standard measure called the Trimmed Mean: https://statisticsbyjim.com/basics/trimmed-mean/ https://statisticsbyjim.com/basics/trimmed-mean/
by mayank 3y ago
This is a pretty standard measure called the Trimmed Mean: https://statisticsbyjim.com/basics/trimmed-mean/ https://statisticsbyjim.com/basics/trimmed-mean/
- stabbles 3y agoFor performance benchmarking the minimal runtime is typically the best estimator if the computations are identical, cause it measures perf w/o interrupts. If the language is garbage collected, or if the test is randomized you obviously don't want to look at the minimum.
- mayank 3y ago> the minimal runtime is typically the best estimator Depends what you’re estimating. The minimum is usually not representative of “real world” performance, which is why we use measures of central tendency over many runs for performance benchmarks.
- ProblemFactory 3y agoVariability in software runtime arises mostly from other software running on the same system. If you are looking for a real-world, whole-system benchmark (like a database or app server), then taking the average makes sense. If you are benchmarking an individual algorithm or program and its optimisations, then taking the fastest run makes sense - that was the run with least external interference. The only exception might be if you want to benchmark with cold caches, but then you need to reset these carefully between runs as well.