5 ms·

To Clojure-curious people: please ignore this article. It is misinformed. Others on this thread have pointed out that the Alioth benchmark takes startup time i

by gcv 14y ago

To Clojure-curious people: please ignore this article. It is misinformed. Others on this thread have pointed out that the Alioth benchmark takes startup time into account. Yes, this imposes a startup penalty on Clojure. More importantly, the implementations of each individual benchmark vary significantly in performance quality. High-performance Clojure requires a couple of tricks in type hinting, using unchecked arithmetic, and preferring native Java arrays. I looked at a couple of the benchmarks, and the mandelbrot example uses those tricks. Notice the performance there: http://shootout.alioth.debian.org/u64/performance.php?test=mandelbrot http://shootout.alioth.debian.org/u64/performance.php?test=m... Notice that Java 7 and gcc run at about the same speed, and Clojure is only about 2.2x slower (than C). Scala is about 1.9x slower (than C). gcc is about 1.5x slower than Intel Fortran. To make the benchmark more fair: (1) all timings should disregard startup time; and (2) all JVM languages should have the opportunity to run the benchmark a few thousand times before timing it. Otherwise, it measures JVM startup time, and then it measures how long it takes the JIT to achieve maximum optimization. By comparison, the C and Fortran code runs at full speed almost out of the gate. All-in-all, considering that Clojure is an extremely high-level language I consider its performance impressive [1]. Yes, the inner loops need to be coded in a slightly un-idiomatic manner, but you can do all this in the comfort of your REPL, which makes the process of making optimizations reasonably painless. [1] Don't forget to scroll down the mandelbrot results and look at the stellar performance of other popular high-level languages.

igouy 14y ago
>>"Otherwise, it measures JVM startup time, and then it measures how long it takes the JIT to achieve maximum optimization."<< Please take that Clojure mandelbrot program, make repeated timing measurements without restarting the JVM and then report how those times compare to cold start on your computer. The mean "warmed" times for the Java mandelbrot program were actually slower than the reported cold start time for the same program. http://shootout.alioth.debian.org/help.php#java http://shootout.alioth.debian.org/help.php#java
- gcv 14y ago
  Sure thing. I made it run for 4000 cycles twice. The first (cold) run took 2831.323 ms, and the warmed-up run took 2571.397 ms. About 10% faster. Judging by the invocation noted at the bottom of http://shootout.alioth.debian.org/u64/program.php?test=mandelbrot&lang=clojure&id=5 http://shootout.alioth.debian.org/u64/program.php?test=mande..., the comments about Java in the FAQ do not apply to the Clojure code. The benchmark seems to have been invoked straight from the command line.
  - igouy 14y ago
    Cool! Now let's check the basics: - 2831.323 ms for what workload? The benchmarks game measurements are made at 3 different workloads; but the times that matter are those for the largest workload, in this case N=16,000. So please show the times for N=16,000. - the benchmarks game measurements are made with output redirected to /dev/null - both the clojure and java programs are invoked straight from the command line, and include start-up. The Help page provides additional "warmed" measurements for the fastest Java programs, for comparison - because sometimes the JVM startup costs are larger in the mind than they are when measured :-)
    - gcv 14y ago
      Well certainly, in a real-world program which runs for long periods analyzing tons of data, JVM startup costs and JIT costs amount to nothing, as they are paid in the first few seconds. But we are talking about short, synthetic benchmarks here. I ran mandelbrot for 4000 cycles, just invoking the function which does the work twice and wrapping each call in Clojure's (time ...) form. This all happened in an AOT-compiled .class file, which guaranteed a cold start. For what it's worth, I didn't bother tuning the GC or any other JVM parameters — I suspect I could have made it run a bit faster by manipulating generation sizes.
      - igouy 14y ago
        >>I ran mandelbrot for 4000 cycles<< That reduced workload only runs the program for 1/10th the time of the workload shown on the benchmarks game website. Run the program for N=16,000 and see that "JVM startup costs and JIT costs amount to nothing" even for these "short, synthetic benchmarks". (Incidentally, the "usual" cold start measurement shown on the website is the best of 6.)

by gcv 14y ago

igouy 14y ago

>>"Otherwise, it measures JVM startup time, and then it measures how long it takes the JIT to achieve maximum optimization."<< Please take that Clojure mandelbrot program, make repeated timing measurements without restarting the JVM and then report how those times compare to cold start on your computer. The mean "warmed" times for the Java mandelbrot program were actually slower than the reported cold start time for the same program. http://shootout.alioth.debian.org/help.php#java http://shootout.alioth.debian.org/help.php#java

gcv 14y ago
Sure thing. I made it run for 4000 cycles twice. The first (cold) run took 2831.323 ms, and the warmed-up run took 2571.397 ms. About 10% faster. Judging by the invocation noted at the bottom of http://shootout.alioth.debian.org/u64/program.php?test=mandelbrot&lang=clojure&id=5 http://shootout.alioth.debian.org/u64/program.php?test=mande..., the comments about Java in the FAQ do not apply to the Clojure code. The benchmark seems to have been invoked straight from the command line.
- igouy 14y ago
  Cool! Now let's check the basics: - 2831.323 ms for what workload? The benchmarks game measurements are made at 3 different workloads; but the times that matter are those for the largest workload, in this case N=16,000. So please show the times for N=16,000. - the benchmarks game measurements are made with output redirected to /dev/null - both the clojure and java programs are invoked straight from the command line, and include start-up. The Help page provides additional "warmed" measurements for the fastest Java programs, for comparison - because sometimes the JVM startup costs are larger in the mind than they are when measured :-)
  - gcv 14y ago
    Well certainly, in a real-world program which runs for long periods analyzing tons of data, JVM startup costs and JIT costs amount to nothing, as they are paid in the first few seconds. But we are talking about short, synthetic benchmarks here. I ran mandelbrot for 4000 cycles, just invoking the function which does the work twice and wrapping each call in Clojure's (time ...) form. This all happened in an AOT-compiled .class file, which guaranteed a cold start. For what it's worth, I didn't bother tuning the GC or any other JVM parameters — I suspect I could have made it run a bit faster by manipulating generation sizes.
    - igouy 14y ago
      >>I ran mandelbrot for 4000 cycles<< That reduced workload only runs the program for 1/10th the time of the workload shown on the benchmarks game website. Run the program for N=16,000 and see that "JVM startup costs and JIT costs amount to nothing" even for these "short, synthetic benchmarks". (Incidentally, the "usual" cold start measurement shown on the website is the best of 6.)