3 ms·

It's an extremely simple algorithm, just one loop with an iterated expression inside it. You can check the source code at: https://github.com/niklas-heer/speed

by ajross 10mo ago

It's an extremely simple algorithm, just one loop with an iterated expression inside it. You can check the source code at: https://github.com/niklas-heer/speed-comparison/tree/master/src https://github.com/niklas-heer/speed-comparison/tree/master/... It's true this is a microbenchmark and not super informative about "Big Problems" (because nothing is). But it absolutely shows up code generation and interpretation performance in an interesting way. Note in particular the huge delta between rust 1.92 and nightly. I'm gonna guess that's down to the autovectorizer having a hole that the implementation slipped through, and they fixed it.

Aurornis 10mo ago
> Note in particular the huge delta between rust 1.92 and nightly. I'm gonna guess that's down to the autovectorizer having a hole that the implementation slipped through, and they fixed it. The benchmark also includes startup time, file I/O, and console printing. There could have been a one-time startup cost somewhere that got removed. The benchmark is not really testing the Leibniz loop performance for the very fast languages, it's testing startup, I/O, console printing, etc.
- igouy 10mo ago
  Let's compare some independent measurements with in-process measurements of tiny tiny nbody programs: https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbodyinprocess.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... How significant are the differences for such tiny tiny programs?
pjscott 10mo ago
The delta there is because the Rust 1.92 version uses the straightforward iterative code and the 1.94-nightly version explicitly uses std::simd vectorization. Compare the source code: https://github.com/niklas-heer/speed-comparison/blob/master/src/leibniz.rs https://github.com/niklas-heer/speed-comparison/blob/master/... https://github.com/niklas-heer/speed-comparison/blob/master/src/leibniz_nightly.rs https://github.com/niklas-heer/speed-comparison/blob/master/...
Hizonner 10mo ago
Take their Haskell implementation. If I compile it unoptimized (with a slight change to hardwire the iteration count at one billion instead of reading it from a file), I get bored waiting for it to finish after several minutes of clock time. If I compile it with "-O3", it runs in 4.75 seconds (it's an old machine). Suppose I remove the strictness annotations (3 exclamation points, in places that aren't obvious to a naive programmer coming from almost any other language). If I then compile it unoptimized, it gets up to over 30GB of resident memory before I get bored (it's an old machine with a lot of memory). It would probably die with an out of memory error if I tried to run it to completion. However, if I compile that same modified code optimized, the compiler infers the strictness and the program runs in exactly the same time as it does with the annotations there. BUT it's far from obvious to the casual observer when the compiler can make those inferences and when it can't. I had ChatGPT rewrite the Haskell code to use unboxed numeric types. It ran in 1.5 seconds (the C version takes 1.27). The rewrite mostly consists of sprinkling "#" liberally throughout the code, but also requires using a few specialized functions. I had ChatGPT do it because I have never used unboxed types, and you could argue that they're not common idiom. However, anybody who actually wrote that kind of numerical code in Haskell on a regular basis would use unboxed types as a matter of course. So which one is the right time?
- ajross 10mo ago
  > So which one is the right time? You're acting like this is a gotcha, but the answer is obviously "all of them" and that indeed, this tells you interesting things about the behavior of your compiler. There are lots of variant scores in the linked article that reflect different ways of expressing the problem. But also, it tells you something about the limitations of your language too. For example, the biggest single reason that C/C++ (and languages like Fortran/Zig/D and sometimes C# and Rust whose code generation is isomorphic to them) sit at the top of the list is that they autovectorize. SIMD code isn't a common idiom either, but the compiler figures it out anyway. And apparently Haskell isn't capable of doing enough static analysis to fall back to an unboxed implementation (though given the simplicity of this code, that should be possible I think). That's not a "flaw" and it doesn't mean Haskell "loses", but it absolutely is an important thing to note. And charts like this show us where those holes lie.

by ajross 10mo ago

Aurornis 10mo ago

> Note in particular the huge delta between rust 1.92 and nightly. I'm gonna guess that's down to the autovectorizer having a hole that the implementation slipped through, and they fixed it. The benchmark also includes startup time, file I/O, and console printing. There could have been a one-time startup cost somewhere that got removed. The benchmark is not really testing the Leibniz loop performance for the very fast languages, it's testing startup, I/O, console printing, etc.

igouy 10mo ago
Let's compare some independent measurements with in-process measurements of tiny tiny nbody programs: https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbodyinprocess.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... How significant are the differences for such tiny tiny programs?

pjscott 10mo ago

The delta there is because the Rust 1.92 version uses the straightforward iterative code and the 1.94-nightly version explicitly uses std::simd vectorization. Compare the source code: https://github.com/niklas-heer/speed-comparison/blob/master/src/leibniz.rs https://github.com/niklas-heer/speed-comparison/blob/master/... https://github.com/niklas-heer/speed-comparison/blob/master/src/leibniz_nightly.rs https://github.com/niklas-heer/speed-comparison/blob/master/...

Hizonner 10mo ago

Take their Haskell implementation. If I compile it unoptimized (with a slight change to hardwire the iteration count at one billion instead of reading it from a file), I get bored waiting for it to finish after several minutes of clock time. If I compile it with "-O3", it runs in 4.75 seconds (it's an old machine). Suppose I remove the strictness annotations (3 exclamation points, in places that aren't obvious to a naive programmer coming from almost any other language). If I then compile it unoptimized, it gets up to over 30GB of resident memory before I get bored (it's an old machine with a lot of memory). It would probably die with an out of memory error if I tried to run it to completion. However, if I compile that same modified code optimized, the compiler infers the strictness and the program runs in exactly the same time as it does with the annotations there. BUT it's far from obvious to the casual observer when the compiler can make those inferences and when it can't. I had ChatGPT rewrite the Haskell code to use unboxed numeric types. It ran in 1.5 seconds (the C version takes 1.27). The rewrite mostly consists of sprinkling "#" liberally throughout the code, but also requires using a few specialized functions. I had ChatGPT do it because I have never used unboxed types, and you could argue that they're not common idiom. However, anybody who actually wrote that kind of numerical code in Haskell on a regular basis would use unboxed types as a matter of course. So which one is the right time?

ajross 10mo ago
> So which one is the right time? You're acting like this is a gotcha, but the answer is obviously "all of them" and that indeed, this tells you interesting things about the behavior of your compiler. There are lots of variant scores in the linked article that reflect different ways of expressing the problem. But also, it tells you something about the limitations of your language too. For example, the biggest single reason that C/C++ (and languages like Fortran/Zig/D and sometimes C# and Rust whose code generation is isomorphic to them) sit at the top of the list is that they autovectorize. SIMD code isn't a common idiom either, but the compiler figures it out anyway. And apparently Haskell isn't capable of doing enough static analysis to fall back to an unboxed implementation (though given the simplicity of this code, that should be possible I think). That's not a "flaw" and it doesn't mean Haskell "loses", but it absolutely is an important thing to note. And charts like this show us where those holes lie.