4 ms·
Great article! Statistical fallacies are rampant in performance eval, even in academic settings. When designing statistical tests for performance, the keyword
by ericpauley 4y ago
Great article!
Statistical fallacies are rampant in performance eval, even in academic settings. When designing statistical tests for performance, the keyword you want to use here is non-parametric. I.e., a U-test is a non-parametric analog to the t-test. It just looks at the rank statistics of results instead of their value, thus eliminating dependence on the underling distribution.
Another issue that pops up is sample independence. Statistical tests are often predicated on each sample being independent and identically distributed (i.i.d.), but in reality this is often not the case. For instance, running all the tests of one group and then all the tests of the other could heat the CPU and cause reduced performance in the second trial.
- zX41ZdbW 4y agoWe use non-parametric statistics for performance testing in ClickHouse[1], picked from the article "A Randomized Design Used in the Comparison of Standard and Modified Fertilizer Mixtures for Tomato Plants". [1] https://clickhouse.com/blog/testing-the-performance-of-click-house https://clickhouse.com/blog/testing-the-performance-of-click...
- kurthr 4y agoMany would do well to look at simple DOE when building experimental design. At least look at the data distributions before taking means and stdev. NIST has some nice simple descriptions and example experimental designs: https://www.itl.nist.gov/div898/handbook/pmd/pmd.htm