3 ms·

Am I reading that right? There is a whole article about how long JIT compilers take to settle, but the actual performance changes by just a few percent tops? (f

by theamk 8y ago

Am I reading that right? There is a whole article about how long JIT compilers take to settle, but the actual performance changes by just a few percent tops? (for example, first graph has 0.571 for cold JIT, and 0.568 for warm JIT, a 0.5% difference) I think for me, the main takeaway from article is startup time for JITs is irrelevant, since it does not affect performance much.

jsnell 8y ago
That was my reading of the graphs as well, and I was surprised it wasn't addressed in the text.
ltratt 8y ago
In general, yes, you're reading the timings right, but there's a couple of factors to consider. First, the JIT compiler has often done most of its work during the first in-process iteration, so the plots are not generally following a "one in-process iteration is slow, then the JIT kicks in from in-process iteration 2 onwards" model. Put another way, the JIT compiler is often making even the first in-process iteration run pretty fast. Second, I would suggest that the often small timing differences are more worrying than they may first appear. In a semi-mature compiler, optimisations are frequently in the range of a 0.5-1% improvement. So if your measurements are only accurate to (say) 2%, most of your attempted optimisations will be misclassified (bad optimisations will sometimes be measured as good; good optimisations will sometimes be measured as bad). Furthermore, when a VM recompiles things, it generally performs several optimisations at once. Thus, if the overall performance gets worse (even if by a small bit), it may suggest that several optimisations performed badly at the same time.
- Boxxed 8y ago
  But doesn't that mean the article's point is kind of moot? That when people blame slow languages / VMs, they actually are slow and it has nothing to do with warm-up time?
  - codeflo 8y ago
    Both can be true, benchmarking JITs is really hard, but even so, (despite decades of claims to the contrary), managed VMs actually are a bit slower than unsafe native code.
    - wbl 8y ago
      Nothing prevents native code from being safe

by theamk 8y ago

jsnell 8y ago

That was my reading of the graphs as well, and I was surprised it wasn't addressed in the text.

ltratt 8y ago

In general, yes, you're reading the timings right, but there's a couple of factors to consider. First, the JIT compiler has often done most of its work during the first in-process iteration, so the plots are not generally following a "one in-process iteration is slow, then the JIT kicks in from in-process iteration 2 onwards" model. Put another way, the JIT compiler is often making even the first in-process iteration run pretty fast. Second, I would suggest that the often small timing differences are more worrying than they may first appear. In a semi-mature compiler, optimisations are frequently in the range of a 0.5-1% improvement. So if your measurements are only accurate to (say) 2%, most of your attempted optimisations will be misclassified (bad optimisations will sometimes be measured as good; good optimisations will sometimes be measured as bad). Furthermore, when a VM recompiles things, it generally performs several optimisations at once. Thus, if the overall performance gets worse (even if by a small bit), it may suggest that several optimisations performed badly at the same time.

Boxxed 8y ago
But doesn't that mean the article's point is kind of moot? That when people blame slow languages / VMs, they actually are slow and it has nothing to do with warm-up time?
- codeflo 8y ago
  Both can be true, benchmarking JITs is really hard, but even so, (despite decades of claims to the contrary), managed VMs actually are a bit slower than unsafe native code.
  - wbl 8y ago
    Nothing prevents native code from being safe