4 ms·
The problem is that solutions optimised for benchmarks are usually non-idiomatic. If the solution doesn't represent "real code", then is there any real value? A
by icholy 6y ago
The problem is that solutions optimised for benchmarks are usually non-idiomatic. If the solution doesn't represent "real code", then is there any real value? An interesting benchmarking approach would be to take solutions to the same problem from many different programmers at different skill levels and gather metrics from them. I think that would give a much more interesting view into a language's performance characteristics.
- igouy 6y ago> If the solution doesn't represent "real code", then is there any real value? When "idiomatic" performs so poorly that the software is unfit for purpose, does that represent "real code" ? > … solutions to the same problem from many different programmers at different skill levels… How could we know programs weren't written to make some language look bad?
- throwaway894345 6y ago> When "idiomatic" performs so poorly that the software is unfit for purpose, does that represent "real code" ? I'm confused about what argument you're making here. I assume you aren't suggesting that suboptimal performance on a benchmark implies that the software is unfit for purpose? Can you elaborate?
- igouy 6y agoI'm suggesting that "idiomatic" code in real software can perform so poorly that the real software is unfit for purpose. Hence "idiomatic" may represent not "real code" but "naive code".
- throwaway894345 6y agoFair enough, but I don't see how that relates to the conversation. Is that intended to rebut the idea that it's useful to know about the performance of idiomatic software?
- igouy 6y agoYou don't seem to have said "it's useful to know about the performance of idiomatic software".
- throwaway894345 6y agoI’m not the one you were originally responding to but it seemed pretty clear that that’s what they were getting at.
- igouy 6y agoIt seems clear that they conflated "idiomatic" code with "real code" into a subjective dismissal. Do you intend to provide some support for your idea that "it's useful to know about the performance of idiomatic software"?
- throwaway894345 6y ago> It seems clear that they conflated "idiomatic" code with "real code" into a subjective dismissal. No, they're dismissing it as useful for answering the questions we tend to care about when evaluating the performance of various programming languages. One such question is "If I choose a given language, how far will I be able to go with just idiomatic code before I need to invest in expensive optimizations (most of the expense here is that we're spending time optimizing instead of iterating on features)?". "Real code" is a function of this concern, so measuring it doesn't help us answer this question (i.e., if a language performs poorly on this question, "real code" will be more optimized and less idiomatic). Further, microbenchmarks don't measure "real code" either they maximally optimized code, which is what the OP was complaining about.
- igouy 6y ago> One such question… Your question is sensibly answered by — It depends.
- viraptor 6y ago> If the solution doesn't represent "real code", then is there any real value? When you're actually aiming for performance, in my experience code stops looking like "real code" pretty quickly in the hotspot, so benchmarks are representative in that way. I mean, for example media codecs "written in C" are typically highly vectorised, architecture-specific ASM kernels with C around it for handling the flow.