4 ms·

Benchmark sites really need categories like speedruns. The finish line is something like "takes a dynamic input and gets the right answer". Then you have catego

Benchmark sites really need categories like speedruns. The finish line is something like "takes a dynamic input and gets the right answer". Then you have categories that do it under some additional constraints: any% (most languages call out to pre-written machine code, probably) language% (anything possible using just that language) idiomatic%, casual%, singlethreaded%, memory%, compile-run%?, safe%? etc. Then at least we'd be able to argue about which categories a particular benchmark result would apply to, and separately argue about which categories matter in a particular domain.

015a 5y ago
Something like leetcode is the perfect platform for this kind of data; crowdsourced submissions for each language, with automated I/O systems to verify accuracy. In fact, when you submit a solution to a given problem, it tells you your completion time and memory usage percentile relative to other submissions in that language. It'd be very cool if they published statistics for each problem, for each language, but I'm not seeing that data right now.
- davidkunz 5y ago
  Indeed, fast code in an idiomatic style doesn't mean it wasn't optimised for that benchmark. The only code I trust is code written by people having no idea about the benchmark.
tialaramex 5y ago
Each speedrun community decides their own rules. The people in the community are best placed to decide what it is exactly they're comparing and why. So there's no comparison from one community to another. But such comparisons are exactly what we'd be talking about for programming languages. Asking Rust programmers to set the rules for what counts as "idiomatic" Perl makes no sense, just like asking Cuphead players what counts as a "major" glitch in Java Minecraft or Portal 2. But if there is no agreement you haven't really improved anything on the measuring front.
- infogulch 5y ago
  You found it, "idiomatic" is the most subjective of all the potential categories I listed. "idiomatic" may not be useful to compare different languages, but it could still be useful for comparing libraries and frameworks in a single language. If it did work across languages I would expect Perl programmers to decide amongst themselves what is idiomatic, and be willing to explain their reasoning to others (including Rust programmers). But now that we're arguing about specific categories there's a chance that benchmark comparisons could be useful. Right now the whole enterprise is a cluster; categories aren't perfect but they would be an improvement.
  - tialaramex 5y ago
    See, with speed runs we're comparing players - people - on the charts. There is no chart about Grand Poo World 2 versus a Quake Total Conversion, the chart is like, OK, Poo is very good at GPW2 and Panga isn't as fast, but Tofu is faster than Poo. And I can believe a chart could tell me Bob the C++ programmer is better than Jim the C++ programmer at some task a community of C++ programmers agreed on. But can it tell me that Bob the C++ programmer is better than Sarah the C programmer? Or more so, that C++ is better than C? No, the C++ and C communities are going to disagree about how to measure this.
Mathnerd314 5y ago
> language% (anything possible using just that language) Even a category like this is subject to debate, because of intrinsics. C++ compilers provide many "intrinsic" functions that map to machine code instructions, but these functions aren't defined in the standard and the names can vary between compilers. So are they part of the language? ¯\_(ツ)_/¯

015a 5y ago

Something like leetcode is the perfect platform for this kind of data; crowdsourced submissions for each language, with automated I/O systems to verify accuracy. In fact, when you submit a solution to a given problem, it tells you your completion time and memory usage percentile relative to other submissions in that language. It'd be very cool if they published statistics for each problem, for each language, but I'm not seeing that data right now.

davidkunz 5y ago
Indeed, fast code in an idiomatic style doesn't mean it wasn't optimised for that benchmark. The only code I trust is code written by people having no idea about the benchmark.

tialaramex 5y ago

Each speedrun community decides their own rules. The people in the community are best placed to decide what it is exactly they're comparing and why. So there's no comparison from one community to another. But such comparisons are exactly what we'd be talking about for programming languages. Asking Rust programmers to set the rules for what counts as "idiomatic" Perl makes no sense, just like asking Cuphead players what counts as a "major" glitch in Java Minecraft or Portal 2. But if there is no agreement you haven't really improved anything on the measuring front.

infogulch 5y ago
You found it, "idiomatic" is the most subjective of all the potential categories I listed. "idiomatic" may not be useful to compare different languages, but it could still be useful for comparing libraries and frameworks in a single language. If it did work across languages I would expect Perl programmers to decide amongst themselves what is idiomatic, and be willing to explain their reasoning to others (including Rust programmers). But now that we're arguing about specific categories there's a chance that benchmark comparisons could be useful. Right now the whole enterprise is a cluster; categories aren't perfect but they would be an improvement.
- tialaramex 5y ago
  See, with speed runs we're comparing players - people - on the charts. There is no chart about Grand Poo World 2 versus a Quake Total Conversion, the chart is like, OK, Poo is very good at GPW2 and Panga isn't as fast, but Tofu is faster than Poo. And I can believe a chart could tell me Bob the C++ programmer is better than Jim the C++ programmer at some task a community of C++ programmers agreed on. But can it tell me that Bob the C++ programmer is better than Sarah the C programmer? Or more so, that C++ is better than C? No, the C++ and C communities are going to disagree about how to measure this.

Mathnerd314 5y ago

> language% (anything possible using just that language) Even a category like this is subject to debate, because of intrinsics. C++ compilers provide many "intrinsic" functions that map to machine code instructions, but these functions aren't defined in the standard and the names can vary between compilers. So are they part of the language? ¯\_(ツ)_/¯