4 ms·
As someone who contributed to this benchmark it really isn't a great one. First of the JSON parsing is done before the timer starts. If you look at the raw resu
by PMunch 3y ago
As someone who contributed to this benchmark it really isn't a great one. First of the JSON parsing is done before the timer starts. If you look at the raw results you can see that the full runtime of the Julia solution for 60k posts is actually 9.6 seconds (compared to 3.7, 3.8, 4.4, and 7.9 seconds for Rust, Go, Zig, and Nim respectively). The only thing it really measures is hashing and hash table performance (with a sprinkling of GC and memory management). And of course with any benchmark like this how much dev time each respective language put into the benchmark.
- mcronce 3y agoThey're also benchmarking on Github Actions runners, with swings of up to ~50% from run to run, which is more than enough to shuffle the results to more or less random positions. I contributed to it last week but without any will to solve that kind of fundamental problem I don't see it as being particularly good. There's also no control on quality of contributions to the language-specific benchmarks.
- eigenspace 3y agoIt's been moved from Github actions to a Azure F4s v2 - 4vCPU-8GB-Ubuntu 22.04. Still not good for reliability in a benchmark though.
- bafe 3y agoIf I understood the code and the GitHub actions well, it also appears that they run each benchmark once. If, as you said GitHub actions runners show that much variability between runs, one should at least run the action multiple times and report the aggregate running time along with other statistics (standard deviation)...
- mcronce 3y agoAccording to another replier it's been moved to dedicated VMs in Azure, so it's not as bad, but still subject to noisy neighbors. I agree with your assessment - if I were fixing it I would do something similar.
- bafe 3y agoI would think it's one of the most basic rules of benchmarking (or so I was thought during my earlier days as a student) that one should repeat the benchmark several time to smooth over the "randomness" inherent in the system
- eigenspace 3y agoThat seems to instead be accounted for in this benchmark by just parsing more entries. The longer running the benchmark (if the task is homogeneous), the less noise should be relevant.
- tedunangst 3y agoYou can land on a VM host that's busy or idle for longer than the benchmark runs, even if you run for hours.
- eigenspace 3y agoYeah of course. But that’d also be affect it if the benchmark was shorter and was re-run a hundred times. Though, granted in the case of re-running it you can do things like take the minimum or median time which are much better benchmark metrics, rather than the mean which is thrown off more by outliers and system noise. Definitely bot trying to defend this as a good benchmarking scheme.
- mrklol 3y agoThey sadly switched from including json time to excluding it… :(
- dralley 3y agoRust uses a particularly paranoid (slow) hashing algorithm by default.
- fyzix 3y agoThe rust solution is not using std hashmap. It uses FXhashmap as recommended by the rust perf book: https://nnethercote.github.io/perf-book/hashing.html https://nnethercote.github.io/perf-book/hashing.html
- fyzix 3y agoThe total time you see there includes jit compilation and a warm up run.