4 ms·

Although this is a potentially exciting development, I dislike when people use phrases like "speedups of up to 20%." Quoting only the best-case scenario is misl

Although this is a potentially exciting development, I dislike when people use phrases like "speedups of up to 20%." Quoting only the best-case scenario is misleading, since it says little about the average case. What if that 20% speedup only occurs 2% of the time, and the other 98% of the time, you have a <2% improvement? I understand that people want to promote and be recognized for their work by referencing the best cases, but it would be more helpful if they quoted typical results, like the range of speedups achieved on the benchmarks.

dfox 18y ago
Interpreter speedups of this kind (and probably almost any kind) tend to not have significant impact on most real-world workloads, but nonetheless are very important in the long run. Thus it is counterproductive to measure impact of these changes on any realistic benchmark, because it will be lost in the noise. And phrase "on various benchmarks" looks adequate for commit message.
- lunchbox 18y ago
  I think this speedup (and any optimization) is important in the long run; I just dislike when numbers are used without being put in context. I see this in the media all the time. Like when they say that taking [X supplement/drug/food] can reduce your risk of cancer by up to 20%, and then you find out in the fine print that it's only if you're over 65, female, over 200 pounds, have a specific rare genetic mutation...
  - dfox 18y ago
    Numbers without context are certainly misleading, but my opinion is that commit message is not right place to specify when such speedup applies, because for expected readers that is either obvious or irrelevant (or both at once). Other question is whenever this patch is really so interesting to HN. It is pretty much insignificant for real applications and I'd consider the computed goto trick well known in circles where it might be useful for something (which means interpretive VMs and not much else).
    - lunchbox 18y ago
      Agreed on both points.
lbrandy 18y ago
Ah, except... there is no such thing as an "average case" or or a "typical" case in computational workloads. If you want to quote a number, the best-case is the only one that has any meaning whatsoever. The next best thing is to "cluster" or qualify the number (ie, "for memory-bound applications") but even that tends to be nearly impossible to generalize. The only case that actually matters is _your_ case and the only way you'll know is to benchmark it yourself.

dfox 18y ago

Interpreter speedups of this kind (and probably almost any kind) tend to not have significant impact on most real-world workloads, but nonetheless are very important in the long run. Thus it is counterproductive to measure impact of these changes on any realistic benchmark, because it will be lost in the noise. And phrase "on various benchmarks" looks adequate for commit message.

lunchbox 18y ago
I think this speedup (and any optimization) is important in the long run; I just dislike when numbers are used without being put in context. I see this in the media all the time. Like when they say that taking [X supplement/drug/food] can reduce your risk of cancer by up to 20%, and then you find out in the fine print that it's only if you're over 65, female, over 200 pounds, have a specific rare genetic mutation...
- dfox 18y ago
  Numbers without context are certainly misleading, but my opinion is that commit message is not right place to specify when such speedup applies, because for expected readers that is either obvious or irrelevant (or both at once). Other question is whenever this patch is really so interesting to HN. It is pretty much insignificant for real applications and I'd consider the computed goto trick well known in circles where it might be useful for something (which means interpretive VMs and not much else).
  - lunchbox 18y ago
    Agreed on both points.

lbrandy 18y ago

Ah, except... there is no such thing as an "average case" or or a "typical" case in computational workloads. If you want to quote a number, the best-case is the only one that has any meaning whatsoever. The next best thing is to "cluster" or qualify the number (ie, "for memory-bound applications") but even that tends to be nearly impossible to generalize. The only case that actually matters is _your_ case and the only way you'll know is to benchmark it yourself.