5 ms·

I don't understand, what are the columns? Is this ratio thing a factor of how much slower one library is than some other unspecified thing? And why aren't all b

by Aachen 3y ago

I don't understand, what are the columns? Is this ratio thing a factor of how much slower one library is than some other unspecified thing? And why aren't all benchmarks run on all engines if the results are averaged anyway; does the benchmark count column mean something different?

Modified3019 3y ago
Yeah, definitely needs at least a “higher/lower is better” comment for people that don’t want to sink time and effort into a random github project trying to figure out what these values are saying.
- deleted 3y ago
  [deleted]
burntsushi 3y ago
Does this paragraph above the summary table answer your question? > The summary statistic used is the geometric mean of the speed ratios for each regex engine across all benchmarks that include it. The ratios within each benchmark are computed from the median of all timing samples taken, and dividing it by the best median of the regex engines that participated in the benchmark. For example, given two regex engines A and B with results 35 ns and 25 ns on a single benchmark, A has a speed ratio of 1.4 and B has a speed ratio of 1.0. The geometric mean reported here is then the "average" speed ratio for that regex engine across all benchmarks. So 1.0 represents the fastest possible ratio, where the engine ranked first in all benchmarks. > And why aren't all benchmarks run on all engines Because they can't be. Some benchmarks, for example, test the speed that a regex engine reports capture group offsets. But not all regex engines (like Hyperscan) support that functionality. In other cases, it's because of various limits that are hit, e.g., "regex is too big." If you look at the individual benchmarks, more information about each can be seen by expanding the details. And if there are engines missing, it should tell you why. Look at the noseyparker benchmark for example[1]. It's absolutely brutal and most engines just time out. [1]: https://github.com/BurntSushi/rebar#noseyparker https://github.com/BurntSushi/rebar#noseyparker
- kzrdude 3y ago
  That was interesting. And the quadratic benchmark listed after that highlights how hyperscan wins in an unfair way - it just implements different semantics, so it's not comparing the same feature. (I see now, after posting, you've mentioned it in another comment.)

by Aachen 3y ago

Modified3019 3y ago

Yeah, definitely needs at least a “higher/lower is better” comment for people that don’t want to sink time and effort into a random github project trying to figure out what these values are saying.

deleted 3y ago
[deleted]

burntsushi 3y ago

Does this paragraph above the summary table answer your question? > The summary statistic used is the geometric mean of the speed ratios for each regex engine across all benchmarks that include it. The ratios within each benchmark are computed from the median of all timing samples taken, and dividing it by the best median of the regex engines that participated in the benchmark. For example, given two regex engines A and B with results 35 ns and 25 ns on a single benchmark, A has a speed ratio of 1.4 and B has a speed ratio of 1.0. The geometric mean reported here is then the "average" speed ratio for that regex engine across all benchmarks. So 1.0 represents the fastest possible ratio, where the engine ranked first in all benchmarks. > And why aren't all benchmarks run on all engines Because they can't be. Some benchmarks, for example, test the speed that a regex engine reports capture group offsets. But not all regex engines (like Hyperscan) support that functionality. In other cases, it's because of various limits that are hit, e.g., "regex is too big." If you look at the individual benchmarks, more information about each can be seen by expanding the details. And if there are engines missing, it should tell you why. Look at the noseyparker benchmark for example[1]. It's absolutely brutal and most engines just time out. [1]: https://github.com/BurntSushi/rebar#noseyparker https://github.com/BurntSushi/rebar#noseyparker

kzrdude 3y ago
That was interesting. And the quadratic benchmark listed after that highlights how hyperscan wins in an unfair way - it just implements different semantics, so it's not comparing the same feature. (I see now, after posting, you've mentioned it in another comment.)