6 ms·

Ordered by realtime, fastest to slowest for those like me who got annoyed by the scrolling up and down trying to compare: Rust (1.13.0-nightly) 1m32.

Ordered by realtime, fastest to slowest for those like me who got annoyed by the scrolling up and down trying to compare: Rust (1.13.0-nightly) 1m32.392s Nim (0.14.2) 1m53.320s C 1m59.116s Julia (0.4.6) 2m01.166s Crystal (0.18.7) 2m01.735s C Double Precision 2m26.546s Java (1.7.0_111) 2m36.949s Nim Double Precision (0.14.2) 3m19.547s OCaml 3m59.597s Go 1.6 6m44.151s node.js (6.2.1) 7m59.041s node.js (5.7.1) 8m49.170s C# 12m18.463s PyPy 14m02.406s Lisp 24m43.216s Haskell 26m34.955s Elixir 123m59.025s Elixir MP 138m48.241s Luajit 225m58.621s Python 348m35.965s Lua 611m38.925s

krull10 8y ago
I wouldn’t give much weight to those benchmark numbers at this point. Some of those language versions are quite out of date...
- kenhwang 8y ago
  I would imagine current versions would only give the newer, actively developed languages a boost like Rust, Nim, Crystal, and Go. Java and C I don't see improving moving much.
  - colordrops 8y ago
    The node version is very old. V8 has improved significantly since v6.
- ilitirit 8y ago
  The numbers are indeed out of date. I re-ran the tests for some of the C and Nim programs using Nim 0.19.4 and gcc 7.3.0 on Windows 10. Here are the results: crb-omp 0m22.949s crb 0m23.240s crb_opt 0m31.404s nimrb_pmap 0m8.828s nimrb_fn 0m22.988s nimrb 0m26.556s The base C code is faster than base Nim code. The optimised C code is significantly slower than everything else(!?). The Nim program that uses a threadpool is the fastest of these. I couldn't get the CPP versions to compile. I'll do the Rust programs some other time.
  - weberc2 8y ago
    It's surprising that the optimized C is slower; it's much faster on my machine. Like 20s (optimized C) vs 1m20s (unoptimized C).
  - tom_mellior 8y ago
    If you are using a current version of your C compiler and can consistently reproduce those numbers just by adding something like -O3, you should probably file a bug report. Optimizers can go wrong and sometimes pessimize things a bit, but an almost 50% slowdown from enabling optimizations would be treated as an important bug to fix. (Though people are saying that significant amounts of time in the benchmark are spent in random number generation and output, so maybe try to find out first if the problem lies in one of those.)
    - tom_mellior 8y ago
      Ah, nevermind, (a) even the "non-optimized" C version uses -O3, and (b) the "C" and the "optimized C" programs differ not only in compiler flags but they are actually different source codes. Specifically, the "optimized C" version doesn't use the faster random number generator. If you fix that, on my machine it's 17.3 seconds for the base C version and 13.4 for the optimized one, i.e., a 22% improvement from turning on the extra optimizations (-march=native and -ffast-math). And for whatever it's worth, because some people love hating on GCC in favor of Clang, my Clang timings are 19.5 and 16.5 seconds, respectively.
weberc2 8y ago
That's a big jump between OCaml and Go. I'm not familiar with ray tracing, but skimming the source code it mostly looks like it's doing floating point math; it doesn't look like it's using the runtime (no allocations, no virtual function calls, no scheduling, etc), so I'm surprised that Go is performing relatively poorly. I wonder if the performance gap is attributable to some overhead in Go's function calls? I know Go passes parameters on the stack instead of via registers... Maybe it's due to passing struct copies instead of references (looks like the C version passes references)? Generally poor code generation? Anyone else have ideas or care to profile? EDIT: From my 2015 MBP, Go (version 1.12) is indeed quite a lot slower than C, but only if you're doing an optimized build `-03`: tmp $ time ./gorb real 1m15.128s user 1m9.366s sys 0m6.754s tmp $ clang crb.c tmp $ time ./a.out real 1m13.041s user 1m10.284s sys 0m0.624s tmp $ gcc crb.c -o crb -std=c11 -O3 -lm -D_XOPEN_SOURCE=600 tmp $ time ./crb real 0m22.703s user 0m22.550s sys 0m0.073s tmp $ clang crb.c -o crb -std=c11 -O3 -lm -D_XOPEN_SOURCE=600 tmp $ time ./crb real 0m22.689s user 0m22.564s sys 0m0.060s EDIT2: I re-modified the Go version (https://gist.github.com/weberc2/2aed4f8d3189d09067d56444836772da https://gist.github.com/weberc2/2aed4f8d3189d09067d564448367...) to pass references and that seems to put it on par with C (or I mistranslated, which is also likely): $ time ./gorb real 0m19.282s user 0m14.467s sys 0m7.523s
- Lerc 8y ago
  I'm not familiar with go but I seem to recall it is garbage collected. If so it may be something to do with the creation of new vectors on the heap instead of the stack. The compiler would have to determine the full lifetime of the vector value to be able to bump it to the stack. That's an optimization, and sometimes it's just not possible (but probably is here). In the C instance no such optimization is necessary. They ask for it on the stack, and if you try to use it after the stack is gone, Bad Things™ happen. A hypothesis to test would be that languages above the jump are managing to work on the stack and ones below are allocating objects on the heap.
  - weberc2 8y ago
    Is it creating vectors in the hot path? I'm not seeing it. Go does some escape analysis (the optimization you're referring to), but it's pretty conservative.
z92 8y ago
Julia was astonishing. It's a high level language that's performing almost like C. Last time I checked, many years back, the spec was changing and the run time did crash. Guess it has gone a long way since. The other one is Lua. My assumption was that it's one of the lightest and fastest language around. Looks like "fastest" isn't true in some cases.
- hackerpacker 8y ago
  shouldn't rust being faster than C be something of a red flag that they aren't quite the same algorithm? Or that the algorithm is sub-optimal?
  - kenhwang 8y ago
    C and Rust have been trading blows on the language benchmark games for a while now which dictates the algorithm used. From my experience, it's relatively easy to accidentally write fast Rust, but incredibly hard to write fast C. https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/gcc-rust.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
  - z92 8y ago
    shouldn't rust being faster than C be something of a red flag that they aren't quite the same algorithm? Or that the algorithm is sub-optimal? The difference isn't much. And Rust is more like FORTRAN. Maybe a bit faster than C, but can't do the gymnastics with pointers that C can.
    - steveklabnik 8y ago
      With unsafe, you can do anything that C can. Without unsafe, there’s significantly more aliasing information, which helps optimizations.
    - oconnor663 8y ago
      > can't do the gymnastics with pointers that C can It can if you write the "unsafe" keyword, but there's a pretty strong community norm around not doing that sort of thing, unless you can encapsulate it inside some sort of safe API. And to be fair to C, I think C can close the gap with Rust/Fortran if you use the "restrict" keyword a lot?
weberc2 8y ago
I rewrote the Go benchmark to be a mechanical translation of C and it performs much better. C (gcc -O3): 23.8s Julia (julia 1.1.0): 32.8s Go (alt) (go 1.12): 39.3s Java (java 1.8.0_60): 44.2s Go (org) (go 1.12): 64.8s OCaml (ocaml 4.07.1): 79.1s JS (node 11.14.0): 137.0s Pypy (pypy 6.0.0): 139.4s C# (mono 4.2.1): 187.3s Rust: DOES NOT COMPILE So Go is only twice as slow as C, not thrice as slow. This puts it just ahead of Java and just behind Julia.
- RandyRanderson 8y ago
  Note that the Java impl is creating objects all over the place in the inner loop - madness! I'm sure a "mechanical translation of [the] C" version would improve things for the Java ver as well. If we removed startup costs (the class file validation, etc) I'd expect it to be on par with C.
  - weberc2 8y ago
    The startup costs are fixed (they don't increase linearly with the number of loop iterations) and for such a small program, they could not reasonbly explain more than 1s of the 20s gap between C and Java. Also, I don't think it's "allocating objects in an inner loop", because Java's allocations are super cheap (bump allocations) if the escape analyzer doesn't keep them on the stack in the first place. That said, after examining this benchmark further, I don't think it's very good since the sequences returned by the random number generators are not controlled for (each implementation uses its own standard library RNG with their own seeds, so the sequences will vary from language to language). This likely causes more loop iterations, but considering the loop termination condition, the theoretical distribution of RNG outputs, and the trivial work done in the loop body, I doubt that the delta in loop iterations can explain any significant portion of the gap. Rather, I think the gap is simply a difference in performance of the RNGs themselves--C and Rust use a poor man's RNG (xorshift) which performs very well for this exercise but is not a good general purpose RNG (and standard library RNGs are optimized for the general case). When I rewrote the Go version, using the xorshift implementation made the most significant impact (15s), although I'm not 100% sure that the output of the RNG isn't just causing it to run the RNG less frequently. I opened up this ticket against the project: https://github.com/niofis/raybench/issues/15 https://github.com/niofis/raybench/issues/15.
Shish2k 8y ago
Python gets 20% faster if you use `__slots__` on the Vector class which is created and destroyed millions of times. It's still the second-slowest, but it's a nice improvement :P

krull10 8y ago

I wouldn’t give much weight to those benchmark numbers at this point. Some of those language versions are quite out of date...

kenhwang 8y ago
I would imagine current versions would only give the newer, actively developed languages a boost like Rust, Nim, Crystal, and Go. Java and C I don't see improving moving much.
- colordrops 8y ago
  The node version is very old. V8 has improved significantly since v6.
ilitirit 8y ago
The numbers are indeed out of date. I re-ran the tests for some of the C and Nim programs using Nim 0.19.4 and gcc 7.3.0 on Windows 10. Here are the results: crb-omp 0m22.949s crb 0m23.240s crb_opt 0m31.404s nimrb_pmap 0m8.828s nimrb_fn 0m22.988s nimrb 0m26.556s The base C code is faster than base Nim code. The optimised C code is significantly slower than everything else(!?). The Nim program that uses a threadpool is the fastest of these. I couldn't get the CPP versions to compile. I'll do the Rust programs some other time.
- weberc2 8y ago
  It's surprising that the optimized C is slower; it's much faster on my machine. Like 20s (optimized C) vs 1m20s (unoptimized C).
- tom_mellior 8y ago
  If you are using a current version of your C compiler and can consistently reproduce those numbers just by adding something like -O3, you should probably file a bug report. Optimizers can go wrong and sometimes pessimize things a bit, but an almost 50% slowdown from enabling optimizations would be treated as an important bug to fix. (Though people are saying that significant amounts of time in the benchmark are spent in random number generation and output, so maybe try to find out first if the problem lies in one of those.)
  - tom_mellior 8y ago
    Ah, nevermind, (a) even the "non-optimized" C version uses -O3, and (b) the "C" and the "optimized C" programs differ not only in compiler flags but they are actually different source codes. Specifically, the "optimized C" version doesn't use the faster random number generator. If you fix that, on my machine it's 17.3 seconds for the base C version and 13.4 for the optimized one, i.e., a 22% improvement from turning on the extra optimizations (-march=native and -ffast-math). And for whatever it's worth, because some people love hating on GCC in favor of Clang, my Clang timings are 19.5 and 16.5 seconds, respectively.

weberc2 8y ago

That's a big jump between OCaml and Go. I'm not familiar with ray tracing, but skimming the source code it mostly looks like it's doing floating point math; it doesn't look like it's using the runtime (no allocations, no virtual function calls, no scheduling, etc), so I'm surprised that Go is performing relatively poorly. I wonder if the performance gap is attributable to some overhead in Go's function calls? I know Go passes parameters on the stack instead of via registers... Maybe it's due to passing struct copies instead of references (looks like the C version passes references)? Generally poor code generation? Anyone else have ideas or care to profile? EDIT: From my 2015 MBP, Go (version 1.12) is indeed quite a lot slower than C, but only if you're doing an optimized build `-03`: tmp $ time ./gorb real 1m15.128s user 1m9.366s sys 0m6.754s tmp $ clang crb.c tmp $ time ./a.out real 1m13.041s user 1m10.284s sys 0m0.624s tmp $ gcc crb.c -o crb -std=c11 -O3 -lm -D_XOPEN_SOURCE=600 tmp $ time ./crb real 0m22.703s user 0m22.550s sys 0m0.073s tmp $ clang crb.c -o crb -std=c11 -O3 -lm -D_XOPEN_SOURCE=600 tmp $ time ./crb real 0m22.689s user 0m22.564s sys 0m0.060s EDIT2: I re-modified the Go version (https://gist.github.com/weberc2/2aed4f8d3189d09067d56444836772da https://gist.github.com/weberc2/2aed4f8d3189d09067d564448367...) to pass references and that seems to put it on par with C (or I mistranslated, which is also likely): $ time ./gorb real 0m19.282s user 0m14.467s sys 0m7.523s

Lerc 8y ago
I'm not familiar with go but I seem to recall it is garbage collected. If so it may be something to do with the creation of new vectors on the heap instead of the stack. The compiler would have to determine the full lifetime of the vector value to be able to bump it to the stack. That's an optimization, and sometimes it's just not possible (but probably is here). In the C instance no such optimization is necessary. They ask for it on the stack, and if you try to use it after the stack is gone, Bad Things™ happen. A hypothesis to test would be that languages above the jump are managing to work on the stack and ones below are allocating objects on the heap.
- weberc2 8y ago
  Is it creating vectors in the hot path? I'm not seeing it. Go does some escape analysis (the optimization you're referring to), but it's pretty conservative.

z92 8y ago

Julia was astonishing. It's a high level language that's performing almost like C. Last time I checked, many years back, the spec was changing and the run time did crash. Guess it has gone a long way since. The other one is Lua. My assumption was that it's one of the lightest and fastest language around. Looks like "fastest" isn't true in some cases.

hackerpacker 8y ago
shouldn't rust being faster than C be something of a red flag that they aren't quite the same algorithm? Or that the algorithm is sub-optimal?
- kenhwang 8y ago
  C and Rust have been trading blows on the language benchmark games for a while now which dictates the algorithm used. From my experience, it's relatively easy to accidentally write fast Rust, but incredibly hard to write fast C. https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/gcc-rust.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- z92 8y ago
  shouldn't rust being faster than C be something of a red flag that they aren't quite the same algorithm? Or that the algorithm is sub-optimal? The difference isn't much. And Rust is more like FORTRAN. Maybe a bit faster than C, but can't do the gymnastics with pointers that C can.
  - steveklabnik 8y ago
    With unsafe, you can do anything that C can. Without unsafe, there’s significantly more aliasing information, which helps optimizations.
  - oconnor663 8y ago
    > can't do the gymnastics with pointers that C can It can if you write the "unsafe" keyword, but there's a pretty strong community norm around not doing that sort of thing, unless you can encapsulate it inside some sort of safe API. And to be fair to C, I think C can close the gap with Rust/Fortran if you use the "restrict" keyword a lot?

weberc2 8y ago

I rewrote the Go benchmark to be a mechanical translation of C and it performs much better. C (gcc -O3): 23.8s Julia (julia 1.1.0): 32.8s Go (alt) (go 1.12): 39.3s Java (java 1.8.0_60): 44.2s Go (org) (go 1.12): 64.8s OCaml (ocaml 4.07.1): 79.1s JS (node 11.14.0): 137.0s Pypy (pypy 6.0.0): 139.4s C# (mono 4.2.1): 187.3s Rust: DOES NOT COMPILE So Go is only twice as slow as C, not thrice as slow. This puts it just ahead of Java and just behind Julia.

RandyRanderson 8y ago
Note that the Java impl is creating objects all over the place in the inner loop - madness! I'm sure a "mechanical translation of [the] C" version would improve things for the Java ver as well. If we removed startup costs (the class file validation, etc) I'd expect it to be on par with C.
- weberc2 8y ago
  The startup costs are fixed (they don't increase linearly with the number of loop iterations) and for such a small program, they could not reasonbly explain more than 1s of the 20s gap between C and Java. Also, I don't think it's "allocating objects in an inner loop", because Java's allocations are super cheap (bump allocations) if the escape analyzer doesn't keep them on the stack in the first place. That said, after examining this benchmark further, I don't think it's very good since the sequences returned by the random number generators are not controlled for (each implementation uses its own standard library RNG with their own seeds, so the sequences will vary from language to language). This likely causes more loop iterations, but considering the loop termination condition, the theoretical distribution of RNG outputs, and the trivial work done in the loop body, I doubt that the delta in loop iterations can explain any significant portion of the gap. Rather, I think the gap is simply a difference in performance of the RNGs themselves--C and Rust use a poor man's RNG (xorshift) which performs very well for this exercise but is not a good general purpose RNG (and standard library RNGs are optimized for the general case). When I rewrote the Go version, using the xorshift implementation made the most significant impact (15s), although I'm not 100% sure that the output of the RNG isn't just causing it to run the RNG less frequently. I opened up this ticket against the project: https://github.com/niofis/raybench/issues/15 https://github.com/niofis/raybench/issues/15.

Shish2k 8y ago

Python gets 20% faster if you use `__slots__` on the Vector class which is created and destroyed millions of times. It's still the second-slowest, but it's a nice improvement :P