9 ms·

Programming language speed comparison using Leibniz formula for π

• forgotpwd16 10mo ago
  Some seeings: - C++ unsurpassable king. - There's a stark jump of times going from ~200ms to ~900ms. (Rust v1.92.0 being an in-between outlier.) - C# gets massive boost (990->225ms) when using SIMD. - But C++ somehow gets slower when using SIMD. - Zig very fast*! - Rust got big boost (630ms->230ms) upgrading v1.92.0->1.94.0. - Nim (that compiles to C then native via GCC) somehow faster than GCC-compiled C. - Julia keeps proving high-level languages can be fast too**. - Swift gets faster when using SIMD but loses much accuracy. - Go fastest language with own compiler (ie not dependent to GCC/LLVM). - V (also compiles to C) expected it (appearing similar) be close to Nim. - Odin (LLVM) & Ada (GCC) surprisingly slow. (Was expecting them to be close to Zig/Fortran.) - Crystal slowest LLVM-based language. - Pure CPython unsurpassable turtle. Curious how D's reference compiler (DMD) compares to the LLVM/GCC front-ends, how LFortran to gfortran, and QBE to GCC/LLVM. Also would like to see Scala Native (Scala currently being inside the 900~1000ms bunch). * Note that uses `@setFloatMode(.Optimized)` which according to docs is equivalent to `--fast-math` but only D/Fortran use this flag (C/C++ do not). ** Uses `@fastmath` AND `@simd`. The comparison supposedly is for performance on idiomatic code and for Julia SIMD is a simple annotation applied to the loop (and Julia may even auto do it) but should still be noted because (as seen in C# example) it can be big.
  • mrsmrtss 10mo ago
    Looking closer at the benchmarks, it seems that C# benchmark is not using AOT, so Go and even Java GraalVM get here an unfair advantage (when looking at the non SIMD versions). There is a non trivial startup time for JIT.
    • mrsmrtss 10mo ago
      Sorry, I can't seem to edit my answer anymore, but I was mistaken, C# version is using AOT. But the are other significant differences here: > var rounds = int.Parse(File.ReadAllText("rounds.txt")); > var pi = 1.0D; > var x = 1.0D; > for (var i = 2; i < rounds + 2; i++) { > x = -x; > pi += x / (2 \* i - 1); > } > pi \*= 4; > Console.WriteLine(pi); For example, if we change the type of 'rounds' variable here from int to double (like it is also in Go version), the code runs significantly faster on my machine.
      • neonsunset 10mo ago
        Try that on ARM64 and the result will be the opposite :) On M4 Max, Go takes 0.982s to run while C# (non-SIMD) and F# are ~0.51s. Changing it to be closer to Go makes the performance worse in a similar manner.
  • neonsunset 10mo ago
    > Go fastest language with own compiler (ie not dependent to GCC/LLVM). C# is using CoreCLR/NativeAOT. Which does not use GCC or LLVM also. Its compiler is more capable than that of Go.
  • Aurornis 10mo ago
    Reading the repo, the benchmark includes the entire program execution from startup to reading the file. For the sub-second compiled languages, it's basically a benchmark of startup times, not performance in the hot loop.
    • igouy 10mo ago
      How much difference would it make for these tiny programs? https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbodyinprocess.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
    • nheer 10mo ago
      Author here. I just tested this with Zig by timing the segments inside the program: 1 billion iterations: - Startup + file read: 0.01ms - Computation: ~200ms - Overhead: 0.01% Even at 1 million iterations (0.24ms total), startup is only 4% overhead. At 1 billion it's essentially zero. The benchmark is definitely measuring the hot loop, not startup time.
  • nheer 10mo ago
    Author here! Thanks for the detailed breakdown. Let me address a few points: - C++ SIMD being slower: The standard C++ uses i & 0x1 which lets the compiler auto-vectorize. With -O3 -ffast-math -march=native, gcc/clang do this really well. The explicit AVX2 version has overhead from manual vector setup and horizontal sum at the end. Modern compilers often beat hand-written SIMD for simple loops like this. - Zig fast-math: Correct. Line 5 has @setFloatMode(.optimized) with a comment saying "like C -ffast-math". - Julia: Also correct. Uses @fastmath @simd for - both annotations together. - Crystal/Odin/Ada being slow: All three use x = -x which creates a loop-carried dependency that blocks auto-vectorization. The fast implementations use the branchless i & 0x1 trick instead. - C# SIMD: Uses Vector512 doing 8 doubles per iteration. That explains the ~4x speedup. - Nim vs C: Both compile via gcc with similar flags. Probably just measurement variance. - Fortran: Interestingly does NOT use -ffast-math. Uses manual loop unrolling instead (processes 4 terms per iteration). - Go: You're right that it's the fastest with its own compiler. No LLVM/GCC backend, just Go's own SSA-based compiler. For suggestions - DMD, LFortran, and Scala Native would be great additions. PRs welcome!
  • vips7L 10mo ago
    > curious how D's reference compiler (DMD) compares to the LLVM/GCC front-ends On my machine (an old i7-8700), dmd performs rather poorly. 3.5 seconds. dmd -O leibniz.d Measure-Command { ./leibniz.exe } | select -expand TotalMilliseconds 3535.823 Comparatively ldc runs in 943 milliseconds: ldmd2 -O leibniz.d Measure-Command { ./leibniz.exe } | select -expand Totalmilliseconds 943.21 I'm sure there is compiler switch magic that I don't know about that could improve them.
• theanonymousone 10mo ago
  Is there a explanation for why C is slower than C++?
  • mutkach 10mo ago
    Probably LLVM runs different sets of optimization passes for C and C++. Need to look at the IR, or assembly to know exactly what happens.
    • pizlonator 10mo ago
      It doesn’t as far as I know. (I have spent a good amount of time hacking the llvm pass pipeline for my personal project so if there was a significant difference I probably would have seen it by now)
      • mutkach 10mo ago
        You are correct, that was an uneducated guess on my part. I just glanced at the IR which was different for some attributes (nounwind vs mustprogress norecurse), but the resulting assembly is 100% identical for every optimization level.
  • AlotOfReading 10mo ago
    It's probably down to the measurement noise of benchmarking on GitHub actions.
    • drob518 10mo ago
      I suspect this is it. Any benchmark that takes less than a second to run should have its iteration count increased such that it takes at least a second, and preferably 5+ seconds, to run. Otherwise CPU scheduling, network processing, etc. is perturbing everything.
      • igouy 10mo ago
        What if instead we measured with … BenchExec "uses the cgroups feature of the Linux kernel to correctly handle groups of processes and uses Linux user namespaces to create a container that restricts interference of [each program] with the benchmarking host." https://github.com/sosy-lab/benchexec https://github.com/sosy-lab/benchexec
• xnacly 10mo ago
  the rust example is so far off being useful and file io seems completly dumb in this context
  • pizlonator 10mo ago
    Real programs have to do IO and the C and C++ code runs faster while also doing IO. What do you think they could have done better assuming that the IO is a necessary part of the benchmark? Also good job to the Rust devs for making the benchmark so much faster in nightly. I wonder what they did.
    • Aurornis 10mo ago
      The file I/O is probably irrelevant, but the startup time is not. The differences among the really fast languages are probably in different startup times if I had to guess.
      • pizlonator 10mo ago
        > differences among the really fast languages are probably in different startup times Startup times matter a great deal.
      • igouy 10mo ago
        What if we try to measure rather than guess :-) https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbodyinprocess.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
    • igouy 10mo ago
      > Assuming that the IO is a necessary part of the benchmark … So make it significant, move giga like benchmarks game reverse-complement & fasta & … https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/revcomp.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
• nmaludy 10mo ago
  If i'm understanding the repository correctly, it looks like each language reads from a file, does some I/O printing to console, then computes the value, then some more console printing and exits. In my opinion, the comparisons could be better if the file I/O and console printing were removed.
  • gavinray 10mo ago
    I'm fairly sure I can speed the JVM implementations up a significant amount by MMAP'ing the file into memory and ensuring it's aligned.
    • pjscott 10mo ago
      I'm not too familiar with the JVM so perhaps I'm missing something here: how would that help? The file is tiny, just a few bytes, so I'd expect the main slowdown to come from system call overhead. With non-mmap file I/O you've got the open/read/close trio, and only one read(2) should be needed, so that's three expensive trips into kernel space. With mmap, you've got open/stat/mmap/munmap/close. Memory-mapped I/O can be great in some circumstances, but a one-time read of a small file is one of the canonical examples for when it isn't worth the hassle and setup/teardown overhead.
    • neonsunset 10mo ago
      [dead]
  • Twirrim 10mo ago
    I'm not sure why the contents of rounds.txt isn't just provided as some kind of argument instead of read in from a file. Given all the other boilerplate involved, I would have expected it to be trivial to add relevant templating.
    • cgh 10mo ago
      Zig could include the file at compile-time with @embedFile.
• arohner 10mo ago
  The Clojure version is not AOT'd, so it's measuring startup + compiler time. When properly compiled it should be comparable to the Java implementation.
• klaff 10mo ago
  I think I get why C++ thru C are all similar (all compile to similar assembly?), but I don't get why Go thru maybe Racket are all in what looks like a pretty narrow clump. Is there a common element there?
  • f1shy 10mo ago
    Some features some of those languages have: - run bytecode - very high level - GC memory But not all have these traits. Not sure.
  • ajross 10mo ago
    I think it's SIMD generation. Managed runtimes have a much harder time autovectorizing, because you can't do any static analysis about things like array sizes. Note that the true low-level tools are all clustered around 2-300ms, and that the next level up are all the "managed" runtimes around 1-2s. The one exception is sort of an exception that proves the rule: it's marked "C# (SIMD)", and looks like a native compiler and not a managed one.
  • Someone 10mo ago
    They’re measuring program execution time, including program startup and tear down. Languages with a more complex runtime take longer for the startup, and all seem to have roughly equally optimized that.
  • pjscott 10mo ago
    The common element is that they're written with the most obvious version of the code, while the ones in the faster bucket are either explicitly vectorized or written in non-obvious ways to help the compiler auto-vectorize. For example, consider the Objective C version of the loop in leibniz.m: for (long i = 2; i <= rounds + 2; i++) { x *= -1.0; pi += x / (2.0 * i - 1.0); } With my older version of Clang, the resulting assembly at -O3 isn't vectorized. Now look at the C version in leibniz.c: rounds += 2u; // do this outside the loop for (unsigned i=2u; i < rounds; ++i) // use ++i instead of i++ { double x = -1.0 + 2.0 * (i & 0x1); // allows vectorization pi += (x / (2u * i - 1u)); // double / unsigned = double } This produces vectorized code when I compile it. When I replace the Objective C loop with that code, the compiler also produces vectorized code. You see something similar in the other kings-of-speed languages. Zig? It's the C code ported directly to a different syntax. D? Exact same. Fortran 90? Slightly different, but still obviously written with compiler vectorization in mind. (For what it's worth, the trunk version of Clang is able to auto-vectorize either version of the loop without help.)
• andrepd 10mo ago
  This is meaningless. The benchmarks are (1) run in github actions, (2) include file and console IO, and (3) are compiled with different compiler flags...
  • tliltocatl 10mo ago
    It is meaningful as an indication of a realistic developer setup rather than a fine-tuned setup you'll only see in a HPC context.
    • pizlonator 10mo ago
      Exactly. Also, winners don’t make excuses. (Not even being snarky. You have to spiritually accept that as a fact if you are in the PL perf game.)
    • andrepd 10mo ago
      It's certainly not meaningful of anything in the first place, since it only tests codegen for a sequence of floating point calculations, which is hardly a representative workload for anything. Even if it were, using an unreliable virtualised environment only introduces noise, especially in the presence of syscalls... So it's worthless even if the entire purpose was to evaluate how fast floating point sums and divisions run in a CI server x)
• Hizonner 10mo ago
  This sort of thing is pretty meaningless unless the code is all written by people who know how to get performance out of their languages (and they're allowed to do so). Did you use the right representation of the data? Did you use the right library? Did you use the right optimization options? Did you choose the fast compiler or the slow one? Did you know there was a faster or slower one? If you're using fancy stuff, did you use it right? I did the same sort of thing with the Seive of Eratosthenes once, on a smaller scale. My Haskell and Python implementations varied by almost a factor of 4 (although you could argue that I changed the algorithm too much on the fastest Python one). OK, yes, all the Haskell ones were faster than the fastest Python one, and the C one was another 4 times faster than the fastest Haskell one... but they were still over the place.
  • ajross 10mo ago
    It's an extremely simple algorithm, just one loop with an iterated expression inside it. You can check the source code at: https://github.com/niklas-heer/speed-comparison/tree/master/src https://github.com/niklas-heer/speed-comparison/tree/master/... It's true this is a microbenchmark and not super informative about "Big Problems" (because nothing is). But it absolutely shows up code generation and interpretation performance in an interesting way. Note in particular the huge delta between rust 1.92 and nightly. I'm gonna guess that's down to the autovectorizer having a hole that the implementation slipped through, and they fixed it.
    • Aurornis 10mo ago
      > Note in particular the huge delta between rust 1.92 and nightly. I'm gonna guess that's down to the autovectorizer having a hole that the implementation slipped through, and they fixed it. The benchmark also includes startup time, file I/O, and console printing. There could have been a one-time startup cost somewhere that got removed. The benchmark is not really testing the Leibniz loop performance for the very fast languages, it's testing startup, I/O, console printing, etc.
      • igouy 10mo ago
• drob518 10mo ago
  Startup time doesn’t seem to be factored in correctly, so any language that uses a bytecode (e.g. Java) or is compiling from source (e.g. Ruby, Python, etc.) will look poor on this. If the kids of applications that you write are ones that exit after a fraction of a second, then sure, this will tell you something. But if you’re writing server apps that run for days/weeks/months, then this is useless.
  • deleted 10mo ago
    [deleted]
  • vhdd 10mo ago
    Python took 86 seconds, if I'm reading it correctly. I can see your point holding for a language like Java, but most of Python's time spent cannot have been startup time, but actual execution time.
    • drob518 10mo ago
      Yea, for Python it’s also just slow (not the language, but CPython; you can do much better with PyPy).
• kstrauser 10mo ago
  How I love pypy for certain tasks. On my laptop: ᐅ time uv run -p cpython-3.14 leibniz.py 3.1415926525880504 ________________________________________________________ Executed in 38.24 secs fish external usr time 37.91 secs 158.00 micros 37.91 secs sys time 0.16 secs 724.00 micros 0.16 secs ᐅ time uv run -p pypy leibniz.py 3.1415926525880504 ________________________________________________________ Executed in 1.52 secs fish external usr time 1.16 secs 0.25 millis 1.16 secs sys time 0.02 secs 1.29 millis 0.02 secs It was a free 25x speedup.
  • eigenspace 10mo ago
    Sure, but at the end of the day, youre still just polishing a turd, and you give up a LOT of ecosystem just to get a still deeply unimpressive benchmark time.
• kiriberty 10mo ago
  And the winner is (Drumroll)... Python - the most popular language in the AI world
  • empiricus 10mo ago
    Well, python for AI is just the syntactic sugar to call pytorch cuda code on the gpu.
• viktorcode 10mo ago
  After seeing Swift's result I had to look into the source to confirm that yes, it was not written by someone who knows the language. But this is a good benchmark results that demonstrate what performance level can you expect from every language when someone not versed in it does the code porting. Fair play
  • pizlonator 10mo ago
    What do you think they could have done better in the Swift code?
    • viktorcode 10mo ago
      Using overflow operators instead of the ones that check for that each iteration.
      • Someone 10mo ago
        Reading https://github.com/niklas-heer/speed-comparison/blob/master/src/leibniz.swift https://github.com/niklas-heer/speed-comparison/blob/master/..., I think the only overflow checks could be in for i in 2...rounds+2 and I would hope/expect the compiler to be smart enough to know that it only has to check “rounds+2” once there. Swift isn’t exactly new anymore, and it’s supported by a large company. What do I overlook?
        • pizlonator 10mo ago
          There’s no way overflow checks are responsible for that enormous speed difference from C
        • deleted 10mo ago
          [deleted]
  • igouy 10mo ago
    Even in verbose Java it's barely 20 lines. Makes the benchmarks game 100 lines seem like major apps. https://benchmarksgame-team.pages.debian.net/benchmarksgame/box-plot-summary-charts.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/comparable.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
• henning 10mo ago
  Some implementations seem vectorization-friendly like the C one that uses a bit-twiddling trick to avoid the `x = -x` line that the Odin implementation and others have. When you put these programs into Godbolt to see what's going on with them, so much of the code is just the I/O part that it's annoying to analyze
• Qem 10mo ago
  It appears Raku runtime improve a lot. It used to end last in comparisons like that, by a large margin, and now is surpassing Perl and CPython.
• systems 10mo ago
  why is ocaml so low, didnt expect this
  • pjscott 10mo ago
    As with all the ahead-of-time compiled languages that I checked, the answer is that it generates non-SIMD code for the hot loop. The assembly code I see in godbolt.org isn't bad at all; the compiler just didn't do anything super clever.
• Aurornis 10mo ago
  Reading the fine print, the benchmark is not just the Leibniz formula like it says in the chart title. It also includes file I/O, startup time, and console printing: > Why do you also count reading a file and printing the output? > Because I think this is a more realistic scenario to compare speeds. Which is fine, but should be noted more prominently. The startup time and console printing obviously aren't relevant for something like the Python run, but at the top of the chart where runs are a fraction of a second it probably accounts for a lot of the differences. Running the inner loop 100 times over would have made the other effects negligible. As written, trying to measure millisecond differences between entire programs isn't really useful unless someone has a highly specific use case where they're re-running a program for fractions of a second instead of using a long-running process.
• dvh 10mo ago
  Python: how much is pi? Swift: 3.7 Python: that's incorrect! Swift: yeah, but it's fast!
• sph 10mo ago
  C# wins hands down in the performance / lines of code metric. There is very little superfluous or that cannot be inferred by the compiler here: https://github.com/niklas-heer/speed-comparison/blob/master/src/leibniz.cs https://github.com/niklas-heer/speed-comparison/blob/master/...
  • forgotpwd16 10mo ago
    C# places quite low though (at ~1s). C# (SIMD) that you see towards the top is more complicated: https://github.com/niklas-heer/speed-comparison/blob/master/src/leibniz-simd.cs https://github.com/niklas-heer/speed-comparison/blob/master/.... In your metric winner is Nim: https://github.com/niklas-heer/speed-comparison/blob/master/src/leibniz.nim https://github.com/niklas-heer/speed-comparison/blob/master/... (followed by Julia and D).
    • mrsmrtss 10mo ago
      C# non SIMD (naive non optimized version) is in the same ballbark as other similar GC languages. Nim version is not some naive version also and seem rather specially crafted so it can be vectorized and still looses to C# SIMD.
      • forgotpwd16 10mo ago
        Loses? My comparison is regarding GP's metric perf/lines_of_code. Let m := perf/lines_of_code = 1/(t × lines_of_code) [highest is better], or to make comparison simpler*, m' := 1/m = t × lines_of_code [lowest is better]. Then**: Nim 1672 Julia 3012 D 3479 C# (SIMD) 5853 C# 8919 >Nim version is not some naive version It's direct translation of formula, using `mod` rather `x = -x`. *Rather comparing numbers << 1. **No blank/comment lines. As cloc and similar tools count.
        • neonsunset 10mo ago
          Nim "cheats" in a similar way C and C++ submissions do: -fno-signed-zeros -fno-trapping-math Although arguably these flags are more reasonable than allowing the use of -march=native. Also consider the inherent advantage popular languages have: you don't need to break out to a completely niche language, while achieving high performance. Saying this, this microbenchmark is naive and does not showcase realistic bottlenecks applications would face like how well-optimized standard library and popular frameworks are, whether the compiler deals with complexity and abstractions well, whether there are issues with multi-threaded scaling, etc etc. You can tell this by performance of dynamically typed languages - since all data is defined in scope of a single function, the compiler needs to do very little work and can hide the true cost of using something like Lua (LuaJIT).
• amelius 10mo ago
  That first big jump in the graph, I thought that it must be the divide between auto-gc'd and non auto-gc'd languages. But then I noticed that Rust is on the wrong side of the divide.
• igouy 10mo ago
  iirc Last time I looked at this: public static void main(String[] args) throws FileNotFoundException { Scanner s = new Scanner(new File("rounds.txt")); long rounds = s.nextLong(); s.close(); double sum = 0.0; double flip = -1.0; for (long i = 1; i <= rounds; i++) { flip *= -1.0; sum += flip / (2 * i - 1); } System.out.println(sum * 4.0); } :the measurements changed dramatically if the order was switched something like: sum += flip / (2 * i - 1); flip *= -1.0; YMMV
  • vips7L 10mo ago
    IIRC scanner is also terribly slow.
• gus_massa 10mo ago
  It would be nice to have a different color for languages that give the exact IEEE 754 result. Other languages can be using some kind of fast math. Sorted by accuracy: # Language Accuracy 14 Swift (SIMD) 8.69 9 Fortran 90 9.44 2 C# (SIMD) 9.49 . [All the others] 9.50
  • pklausler 10mo ago
    Is "accuracy" defined anywhere on that page?
    • gus_massa 10mo ago
      From https://github.com/niklas-heer/speed-comparison/blob/master/dagger-poc/scmeta.py https://github.com/niklas-heer/speed-comparison/blob/master/... // [I removed the error checking] def pi_accuracy(value_str): """Calculate accuracy of computed pi value. Returns the number of correct decimal places (higher is better). """ value = float(value_str) # math.pi is available in MicroPython accuracy = 1 - (value / math.pi) return -math.log10(abs(accuracy))
• nheer 10mo ago
  Hi everyone, author here! I’m genuinely blown away by all the interest in what started as a silly little experiment. The project grew way beyond its original scope. My initial curiosity was simply: how could you set up a pipeline to do automatic speed comparisons? I was less interested in the results as a definitive measure and more in the infrastructure challenge itself. But then the interest kept growing. I tried to modernize things, but one thing became quite notable: the difference between a language that gets actively optimized by its community (like Julia) versus one that just sits there unoptimized is striking. Honestly, I got overwhelmed. Managing all those implementations, keeping versions up to date, reviewing contributions—it was a lot. I basically tapped out for about a year. Now I’m back, and with AI assistance, maintaining this has become much more realistic—updating versions, helping optimize implementations, etc. That said, I’m always happy to accept contributions from folks who know their languages better than I do. Thank you all for your interest and the thoughtful discussion!
  • igouy 10mo ago
    > how could you set up a pipeline to do automatic speed comparisons? This I like! > … actively optimized … versus one that just sits there unoptimized is striking. See N=50,000,000 nbody #1 #2 #3 #4 #5 jdk-23 https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbodyinprocess.html#nbodycharts https://benchmarksgame-team.pages.debian.net/benchmarksgame/... > a silly little experiment This I don't like.
    • nheer 10mo ago
      Thank you for your comment. > > a silly little experiment > This I don't like. What do you mean with that? To explain what I meant. I knew that that in the end this was a microbenchmark. It can certainly give you some clue about a language, but it doesn't tell you the whole picture. In the end it tells you how good a language is (or can be) at loops and floating point math. That's what I meant. I hope that makes it clearer.
      • igouy 10mo ago
        femto benchmark ! Take kostya or hanabi1224 or attractivechaos or … as your starting point and do better. https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#maintenance-burden https://benchmarksgame-team.pages.debian.net/benchmarksgame/...