4 ms·

Aren’t JIT languages at a disadvantage since they are benchmarked through the CLI rather than using a benchmarking library to allow JIT to warmup?

jakobnissen 3y ago
Yes, but the author claims the longest JIT warmup is 0.3 seconds, so it's not an important issue in these benchmarks that take several seconds.
- lifthrasiir 3y ago
  I strongly suspect that the author may have confused the JIT warmup (hard to measure, as you need to ensure that the performance figure have reached the stable point) from the startup overhead (easy to measure).
  - vkazanov 3y ago
    ...and many jitted vms do not even reach a stable point at all, there was a big paper on this a couple of years ago.
    - igouy 3y ago
      This? https://tratt.net/laurie/blog/2018/why_arent_more_users_more_happy_with_our_vms_part_1.html https://tratt.net/laurie/blog/2018/why_arent_more_users_more...
      - vkazanov 3y ago
        Yes, this is the one. Time flies! And VM jits didn't change all that much.
- nithril 3y ago
  0.3s is significant for a task that only takes 1.14s
  - igouy 3y ago
    We don't know that OpenJDK was "Some JIT-based language runtimes take up to ~0.3 second to compile and warm-up" Maybe that was PyPy with program times >5 seconds.
borissk 3y ago
What's CLI?
- zogrodea 3y ago
  Presumably command line interface/terminal. I can type `dotnet run` in the terminal and provide some options to runa .NET program for example.
brabel 3y ago
For one-time problem runs, the JIT languages in practice will not be given time to warm up. All that matters for a user is how fast the application is in practice. It's not about "making it fair" for languages, it's about measuring how fast they go from nothing to results. It doesn't make sense to allow "warmup" time for them unless your expected application is a server which for most of the time will be running "warm" (and even then with scalable containers that assumption may not even be true in some cases). For servers, however, what matters is mostly how fast its HTTP library is and how good the async IO is... check Techempower benchmarks for that: https://www.techempower.com/benchmarks/#hw=ph&test=fortune&section=data-r22 https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...
- nithril 3y ago
  From nothing to result rarely happens in real live. I hardly see someone to start/stop a program per unit task (like piping commands).
  - jltsiren 3y ago
    Code compilation is a common example. In many languages, when you do a clean build of a large project, you run a large number of short-lived processes. With command line tools, the dominant design philosophy is that a program completes one task and exits. The tasks may be large or small, and the same tools are often expected to handle both.
igouy 3y ago
Some language implementations have run time startup costs (bytecode verifier), not-required at run time by other language implementations. https://www.oracle.com/java/technologies/security-in-java.html https://www.oracle.com/java/technologies/security-in-java.ht... Why should that run time startup cost be ignored?

jakobnissen 3y ago

Yes, but the author claims the longest JIT warmup is 0.3 seconds, so it's not an important issue in these benchmarks that take several seconds.

lifthrasiir 3y ago
I strongly suspect that the author may have confused the JIT warmup (hard to measure, as you need to ensure that the performance figure have reached the stable point) from the startup overhead (easy to measure).
- vkazanov 3y ago
  ...and many jitted vms do not even reach a stable point at all, there was a big paper on this a couple of years ago.
  - igouy 3y ago
    This? https://tratt.net/laurie/blog/2018/why_arent_more_users_more_happy_with_our_vms_part_1.html https://tratt.net/laurie/blog/2018/why_arent_more_users_more...
    - vkazanov 3y ago
      Yes, this is the one. Time flies! And VM jits didn't change all that much.
nithril 3y ago
0.3s is significant for a task that only takes 1.14s
- igouy 3y ago
  We don't know that OpenJDK was "Some JIT-based language runtimes take up to ~0.3 second to compile and warm-up" Maybe that was PyPy with program times >5 seconds.

borissk 3y ago

What's CLI?

zogrodea 3y ago
Presumably command line interface/terminal. I can type `dotnet run` in the terminal and provide some options to runa .NET program for example.

brabel 3y ago

For one-time problem runs, the JIT languages in practice will not be given time to warm up. All that matters for a user is how fast the application is in practice. It's not about "making it fair" for languages, it's about measuring how fast they go from nothing to results. It doesn't make sense to allow "warmup" time for them unless your expected application is a server which for most of the time will be running "warm" (and even then with scalable containers that assumption may not even be true in some cases). For servers, however, what matters is mostly how fast its HTTP library is and how good the async IO is... check Techempower benchmarks for that: https://www.techempower.com/benchmarks/#hw=ph&test=fortune&section=data-r22 https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...

nithril 3y ago
From nothing to result rarely happens in real live. I hardly see someone to start/stop a program per unit task (like piping commands).
- jltsiren 3y ago
  Code compilation is a common example. In many languages, when you do a clean build of a large project, you run a large number of short-lived processes. With command line tools, the dominant design philosophy is that a program completes one task and exits. The tasks may be large or small, and the same tools are often expected to handle both.

igouy 3y ago

Some language implementations have run time startup costs (bytecode verifier), not-required at run time by other language implementations. https://www.oracle.com/java/technologies/security-in-java.html https://www.oracle.com/java/technologies/security-in-java.ht... Why should that run time startup cost be ignored?