3 ms·
Does this paragraph above the summary table answer your question? > The summary statistic used is the geometric mean of the speed ratios for each regex engine
by burntsushi 3y ago
Does this paragraph above the summary table answer your question?
> The summary statistic used is the geometric mean of the speed ratios for each regex engine across all benchmarks that include it. The ratios within each benchmark are computed from the median of all timing samples taken, and dividing it by the best median of the regex engines that participated in the benchmark. For example, given two regex engines A and B with results 35 ns and 25 ns on a single benchmark, A has a speed ratio of 1.4 and B has a speed ratio of 1.0. The geometric mean reported here is then the "average" speed ratio for that regex engine across all benchmarks.
So 1.0 represents the fastest possible ratio, where the engine ranked first in all benchmarks.
> And why aren't all benchmarks run on all engines
Because they can't be. Some benchmarks, for example, test the speed that a regex engine reports capture group offsets. But not all regex engines (like Hyperscan) support that functionality. In other cases, it's because of various limits that are hit, e.g., "regex is too big."
If you look at the individual benchmarks, more information about each can be seen by expanding the details. And if there are engines missing, it should tell you why. Look at the noseyparker benchmark for example[1]. It's absolutely brutal and most engines just time out.
[1]: https://github.com/BurntSushi/rebar#noseyparker https://github.com/BurntSushi/rebar#noseyparker
- kzrdude 3y agoThat was interesting. And the quadratic benchmark listed after that highlights how hyperscan wins in an unfair way - it just implements different semantics, so it's not comparing the same feature. (I see now, after posting, you've mentioned it in another comment.)