I tend to think of these things in terms of some limit approaching reality, whatever that is, as the benchmarks continue. If you have an open competition, and t

I tend to think of these things in terms of some limit approaching reality, whatever that is, as the benchmarks continue. If you have an open competition, and there's enough editing and revision by enough actors over time, it gives you some idea of how fast something is likely to be in practice. That open scrutiny and process of suggested revision by experts in a language is a rough sampling of what is likely to be encountered in practice, with a bunch of experts chipping away at the problem. If a difference in times exists, that's about as good as you're likely to get, with further time differences decreasing in size and probability as the whole process continues. I'd even probably argue that even if you have a bunch of naive implementations, it still tells you about what's likely to be encountered in practice, because those naive implementations are often not so naive, and many programmers are more naive than you would like to be the case. Finally, I think in a lot of cases, it's clearer how to do a certain benchmark in a certain language than the essay assumes. Sure, a complex one probably might require some unusual level of expertise. But many benchmarks are fairly small and circumscribed, for better or worse, and consequently pretty tractable.