4 ms·

The speed tests appear to include the startup time of the virtual machine. We all know Java is slow to start up and Go is fast in this respect. That's not use

The speed tests appear to include the startup time of the virtual machine. We all know Java is slow to start up and Go is fast in this respect. That's not useful information. Rather than bias the results this way, much more interesting would be to test what the speeds are (1) after the VMs have launched, and also (2) when the VMs are hot, that is, Java has compiled your Lisp classes.

ledgerdev 4y ago
He does mention that he is looking for language to use for cli with fast startup. For cli and serverless cold start times become important so much so that clojure (non Graal complied) isn't usable for those scenarios.
- deleted 4y ago
  [deleted]
- igouy 4y ago
  So why compare to Clojure, Babashka, Joker?
iampims 4y ago
startup time is key to being able to write one-off programs/tasks and use them in workflows. It doesn’t matter as much for long lived programs, but I don’t think it’s fair to frame this as “bias the results”
- igouy 4y ago
  In which case, shouldn't we expect any comparison to be against language implementations that have excellent startup time?
tmountain 4y ago
Yup, once hotspot kicks in on the JVM, a toy language is going to have a hard time competing with that performance. It’s definitely disingenuous to include JVM start up time in any benchmark. I used to run clojure benchmarks in a REPL and I would watch as the functions I was testing dramatically increased in performance over a few invocations. JIT compilation is pretty incredible.
- dunefox 4y ago
  > It’s definitely disingenuous to include JVM start up time in any benchmark. It's not since startup time matters for CLI applications.
  - pjmlp 4y ago
    Easy, use JIT caches.
- DeathArrow 4y ago
  >JIT compilation is pretty incredible. Not if you compare to AOT for languages who lack garbage collectors. C, C++, Fortran would still be quite faster.
  - pjmlp 4y ago
    JIT cache is the answer, or one of the AOT compilers that exist for ages.
  - kaba0 4y ago
    Given sufficient time and a sufficiently expert programmer. It is not that easy to beat Java for complex programs where your workload is really variable and can’t use the usual tools like arenas.
  - tsimionescu 4y ago
    There are some workloads where Java's JIT can beat C, C++, Fortran AOT compilation. They are probably not very common, but stuff like un-inlining can really save on instruction cache misses in workloads where the most likely path is consistent for fixed periods of time, but changes relatively often (for something like `if COND { foo(); } else { bar(); }` in a hot loop, where COND is sometimes true for a while than false for another while). Of course, Java also has a lot of overhead from the language and implementation level, because of things like no generic lists of integers, no struct embedding leading to lots of pointer-chasing, a tendency for over-allocation, and direct overhead of the advanced JIT itself - performance counters, dummy instructions to support de/re-optimizations on the fly etc.
lawl 4y ago
I can just as well claim that ignoring startup with a small hot loop would unfairly favor a JIT compiler, since it ignores problems that would crop up in real world scenarios such as poor code locality. Basically what you're suggesting would be the absolute best-case for JIT.
pjmlp 4y ago
We all know that Java is slow to start up, because many don't learn how to use JIT caches or AOT solutions that exist for ages.
- didibus 4y ago
  I've used native compilation with Graal, but everytime I tried a cache option or some of the OpenJDK AOT I couldn't figure it out. Can you link to something or explain how to use JIT caches?
  - pjmlp 4y ago
    Sure, with Open JDK https://docs.oracle.com/en/java/javase/18/vm/class-data-sharing.html https://docs.oracle.com/en/java/javase/18/vm/class-data-shar... Or how to use it alongside Spring, https://medium.com/@toparvion/appcds-for-spring-boot-applications-first-contact-6216db6a4194 https://medium.com/@toparvion/appcds-for-spring-boot-applica... For OpenJ9, https://developer.ibm.com/articles/optimize-jvm-startup-with-eclipse-openjj9/ https://developer.ibm.com/articles/optimize-jvm-startup-with... On Android, ART does it automatically since Android 7, https://source.android.com/devices/tech/dalvik/jit-compiler https://source.android.com/devices/tech/dalvik/jit-compiler
    - didibus 4y ago
      Ya, the AppCD for OpenJDK is what I was trying last time, I think on JDK 11, and frankly I couldn't figure it out, like you have to run with a command, dump something to some path, then re-run with specifying the same path and I don't know it didn't seem it was working last time and it was pretty cumbersome.
Jach 4y ago
Eh, the JVM startup time is much less these days (read: these last several years), however loading Clojure is quite slow comparatively and that's what ends up dominating. On my machine, here's a hastily written Fact.java: import java.math.BigInteger; public class Fact { public static BigInteger fact(BigInteger n) { if (n.equals(BigInteger.valueOf(1))) { return BigInteger.valueOf(1); } else { return n.multiply(fact(n.subtract(BigInteger.valueOf(1)))); } } public static void main(String args[]) { System.out.println(fact(BigInteger.valueOf(100))); } } With OpenJDK 11.0.14, `time java Fact.java` gives real/user/sys of 0.267s/0.654s/0.033s. But that's not compiling the file first which some programmers aren't even aware is a thing Java can do now, if I do the usual thing of compiling with javac then on the class file `time java Fact` I instead get 0.050s/0.043s/0.010s. For Clojure, `time java -jar /usr/share/clojure-1.10/lib/clojure.jar fact.clj` gives 0.468s/1.069s/0.056s. Edit: for fun since the author mentioned Common Lisp, `time sbcl --no-sysinit --no-userinit --script fact.lisp` gives 0.005s/0.003s/0.002s. If I make a binary out of this instead, `time ./fact` gives 0.003s/0.002s/0.001s`. Since even a time of an empty C program's a.out can sometimes give numbers like 0.001s/0.000s/0.001s I'm clearly running into precision limits of the time command. (CL itself has a time macro that will helpfully give you a processor cycles count, about 290,000 on my machine for this.) I think the author is overestimating the difficulty of adding features to Common Lisp relative to what they plan to do with this toy; CL for scripting works great too.
- pjmlp 4y ago
  You can also compile it with JIT cache and try a couple of times afterwards, For Java 11, the docs are on https://docs.oracle.com/en/java/javase/11/vm/class-data-sharing.html#GUID-0260F857-A70E-4399-A1DF-A5766BE33285 https://docs.oracle.com/en/java/javase/11/vm/class-data-shar... Latest versions have other improvements like PGO info as well.
  - the_duke 4y ago
    Or use Graal to compile to native.
    - 4y ago
igouy 4y ago
> We all know Java is slow to start up… More specifically, Clojure — https://stuartsierra.com/2019/12/21/clojure-start-time-in-2019 https://stuartsierra.com/2019/12/21/clojure-start-time-in-20...

ledgerdev 4y ago

He does mention that he is looking for language to use for cli with fast startup. For cli and serverless cold start times become important so much so that clojure (non Graal complied) isn't usable for those scenarios.

deleted 4y ago
[deleted]
igouy 4y ago
So why compare to Clojure, Babashka, Joker?

iampims 4y ago

startup time is key to being able to write one-off programs/tasks and use them in workflows. It doesn’t matter as much for long lived programs, but I don’t think it’s fair to frame this as “bias the results”

igouy 4y ago
In which case, shouldn't we expect any comparison to be against language implementations that have excellent startup time?

tmountain 4y ago

Yup, once hotspot kicks in on the JVM, a toy language is going to have a hard time competing with that performance. It’s definitely disingenuous to include JVM start up time in any benchmark. I used to run clojure benchmarks in a REPL and I would watch as the functions I was testing dramatically increased in performance over a few invocations. JIT compilation is pretty incredible.

dunefox 4y ago
> It’s definitely disingenuous to include JVM start up time in any benchmark. It's not since startup time matters for CLI applications.
- pjmlp 4y ago
  Easy, use JIT caches.
DeathArrow 4y ago
>JIT compilation is pretty incredible. Not if you compare to AOT for languages who lack garbage collectors. C, C++, Fortran would still be quite faster.
- pjmlp 4y ago
  JIT cache is the answer, or one of the AOT compilers that exist for ages.
- kaba0 4y ago
  Given sufficient time and a sufficiently expert programmer. It is not that easy to beat Java for complex programs where your workload is really variable and can’t use the usual tools like arenas.
- tsimionescu 4y ago
  There are some workloads where Java's JIT can beat C, C++, Fortran AOT compilation. They are probably not very common, but stuff like un-inlining can really save on instruction cache misses in workloads where the most likely path is consistent for fixed periods of time, but changes relatively often (for something like `if COND { foo(); } else { bar(); }` in a hot loop, where COND is sometimes true for a while than false for another while). Of course, Java also has a lot of overhead from the language and implementation level, because of things like no generic lists of integers, no struct embedding leading to lots of pointer-chasing, a tendency for over-allocation, and direct overhead of the advanced JIT itself - performance counters, dummy instructions to support de/re-optimizations on the fly etc.

lawl 4y ago

I can just as well claim that ignoring startup with a small hot loop would unfairly favor a JIT compiler, since it ignores problems that would crop up in real world scenarios such as poor code locality. Basically what you're suggesting would be the absolute best-case for JIT.

pjmlp 4y ago

We all know that Java is slow to start up, because many don't learn how to use JIT caches or AOT solutions that exist for ages.

didibus 4y ago
I've used native compilation with Graal, but everytime I tried a cache option or some of the OpenJDK AOT I couldn't figure it out. Can you link to something or explain how to use JIT caches?
- pjmlp 4y ago
  Sure, with Open JDK https://docs.oracle.com/en/java/javase/18/vm/class-data-sharing.html https://docs.oracle.com/en/java/javase/18/vm/class-data-shar... Or how to use it alongside Spring, https://medium.com/@toparvion/appcds-for-spring-boot-applications-first-contact-6216db6a4194 https://medium.com/@toparvion/appcds-for-spring-boot-applica... For OpenJ9, https://developer.ibm.com/articles/optimize-jvm-startup-with-eclipse-openjj9/ https://developer.ibm.com/articles/optimize-jvm-startup-with... On Android, ART does it automatically since Android 7, https://source.android.com/devices/tech/dalvik/jit-compiler https://source.android.com/devices/tech/dalvik/jit-compiler
  - didibus 4y ago
    Ya, the AppCD for OpenJDK is what I was trying last time, I think on JDK 11, and frankly I couldn't figure it out, like you have to run with a command, dump something to some path, then re-run with specifying the same path and I don't know it didn't seem it was working last time and it was pretty cumbersome.

Jach 4y ago

Eh, the JVM startup time is much less these days (read: these last several years), however loading Clojure is quite slow comparatively and that's what ends up dominating. On my machine, here's a hastily written Fact.java: import java.math.BigInteger; public class Fact { public static BigInteger fact(BigInteger n) { if (n.equals(BigInteger.valueOf(1))) { return BigInteger.valueOf(1); } else { return n.multiply(fact(n.subtract(BigInteger.valueOf(1)))); } } public static void main(String args[]) { System.out.println(fact(BigInteger.valueOf(100))); } } With OpenJDK 11.0.14, `time java Fact.java` gives real/user/sys of 0.267s/0.654s/0.033s. But that's not compiling the file first which some programmers aren't even aware is a thing Java can do now, if I do the usual thing of compiling with javac then on the class file `time java Fact` I instead get 0.050s/0.043s/0.010s. For Clojure, `time java -jar /usr/share/clojure-1.10/lib/clojure.jar fact.clj` gives 0.468s/1.069s/0.056s. Edit: for fun since the author mentioned Common Lisp, `time sbcl --no-sysinit --no-userinit --script fact.lisp` gives 0.005s/0.003s/0.002s. If I make a binary out of this instead, `time ./fact` gives 0.003s/0.002s/0.001s`. Since even a time of an empty C program's a.out can sometimes give numbers like 0.001s/0.000s/0.001s I'm clearly running into precision limits of the time command. (CL itself has a time macro that will helpfully give you a processor cycles count, about 290,000 on my machine for this.) I think the author is overestimating the difficulty of adding features to Common Lisp relative to what they plan to do with this toy; CL for scripting works great too.

pjmlp 4y ago
You can also compile it with JIT cache and try a couple of times afterwards, For Java 11, the docs are on https://docs.oracle.com/en/java/javase/11/vm/class-data-sharing.html#GUID-0260F857-A70E-4399-A1DF-A5766BE33285 https://docs.oracle.com/en/java/javase/11/vm/class-data-shar... Latest versions have other improvements like PGO info as well.
- the_duke 4y ago
  Or use Graal to compile to native.
  - 4y ago

igouy 4y ago

> We all know Java is slow to start up… More specifically, Clojure — https://stuartsierra.com/2019/12/21/clojure-start-time-in-2019 https://stuartsierra.com/2019/12/21/clojure-start-time-in-20...