7 ms·

So Rust outperforms Nim on all benchmarks? That's kind of expected right?

by jstx1 5y ago

samwillis 5y ago
I think so, but I think the performance gap is a little larger than I was expecting with Nim effectively compiling to C as an intermediary. https://programming-language-benchmarks.vercel.app/rust-vs-c https://programming-language-benchmarks.vercel.app/rust-vs-c https://programming-language-benchmarks.vercel.app/nim-vs-c https://programming-language-benchmarks.vercel.app/nim-vs-c
- MisterTea 5y ago
  > I was expecting with Nim effectively compiling to C as an intermediary. C has nothing to do with performance, runtimes and libraries do. Nim has runtime overhead whereas C and Rust do not.
  - samwillis 5y ago
    Quite true, although my (possibly naive) assumption was that with it being a statically typed language and these being "simple" algorithmic benchmarks the gap would be closer. Not expecting it to match Rust or C in performance, just surprised at the size of the gap. Would be interesting to see what the generated C is like from the Nim code.
    - MisterTea 5y ago
      > Would be interesting to see what the generated C is like from the Nim code. Just did a few quick tests. A hello world in nim is a one liner: echo "hello world" It compiles to a 102136 byte binary on Linux. A hello world c program consisting of a single printf is 16696 bytes.
      - shirleyquirk 5y ago
        https://hookrace.net/blog/nim-binary-size/ https://hookrace.net/blog/nim-binary-size/ gets hello world to parity(17k) with `-d:release --opt:size` and `strip`, and then shows how to get that down to 150 bytes.
      - xigoi 5y ago
        Did you compile it in release mode?
  - arc619 5y ago
    ...both Rust and C have runtimes: https://blog.mgattozzi.dev/rusts-runtime/ https://blog.mgattozzi.dev/rusts-runtime/
  - mratsim 5y ago
    The runtime overhead is type-specific. If you don't use automatically managed types like seq/string and ref, there is no GC underneath. And even then, the GC is just reference counting so no difference from Rust Rc/Arc.
- cogman10 5y ago
  C is an easy intermediary, C is a bad intermediary. The issue with targeting C is you ended up losing a lot of information about intent. Compilers looking at C have to put a lot of trust that what the programmer wrote was intentional (and thus, unoptimizable). Rust and C both end up faster than Nim because they are talking directly to the compiler. IDK enough about Nim to say if it could hit Rust/C speeds if it targeted the LLVM instead, but it would be a lot faster than it currently is targeting C.
  - masklinn 5y ago
    > C is an easy intermediary, C is a bad intermediary. > The issue with targeting C is you ended up losing a lot of information about intent. Compilers looking at C have to put a lot of trust that what the programmer wrote was intentional (and thus, unoptimizable). The same issue exists in C-sourced compiler backends, which is both GCC and LLVM. It’s not exactly a secret that anything which is not strongly exercised by standard C(++) codebases tends to be rather broken for any non-trivial use and takes years to fix (restrict/noalias being the poster child for this, at least in the context of Rust, I think it hasn’t gotten re-disabled yet so it might make a year).
    - cogman10 5y ago
      I think that has less to do with LLVM/GCC being C(++) implementations and more to do with the fact that the primary language compiled by these compilers is C and C++. Other languages that target these compilers aren't used nearly to the same level (maybe swift now?). The JVM, for example, doesn't see the same issue so much with non-Java languages. Mainly because (IMO) it's been hosting non-java languages for several years now.
deepsun 5y ago
It's much better to start improving from the lowest point :)
capableweb 5y ago
> That's kind of expected right? I'm not sure. The following claim is being made on Nim's website: > Modern concepts like zero-overhead iterators and compile-time evaluation of user-defined functions, in combination with the preference of value-based datatypes allocated on the stack, lead to extremely performant code. > Efficient, expressive, elegant > Nim is a statically typed compiled systems programming language A systems programming language need to have exceptionally good performance, so not sure it's to be expected that the performance is so different than Rust. Apparently the Rust versions have received more optimization efforts compared to the Nim code as well, and it seems some people are working on optimizing the Nim versions so we'll see where it lands in the future I guess: https://news.ycombinator.com/item?id=30244188 https://news.ycombinator.com/item?id=30244188
- carlmr 5y ago
  >Apparently the Rust versions have received more optimization efforts compared to the Nim code as well Having programmed in both I think Rust often makes it easier to write fast code by making all the copying vs borrowing more explicit. Nim is obviously still very fast and much easier to pick up, so there are trade-offs here.
- gameswithgo 5y ago
  I've been programming for like 30 years and I've seen dozens of language show up claiming "C like performance" and so far Rust is the only one where that has been true.
  - capableweb 5y ago
    Indeed, probably the most overused selling pitch for programming languages has to be "C like performance", while only providing evidence for a few smaller use cases. I've only dabbled with Rust enough to grok the ownership parts, but I'm happy to hear you feel it lives up to that. As an aside, I'm not expecting a lot of languages to live up to the "C like performance" claim when they make it, as C has been receiving optimizations for over 50 years by now, so it makes sense it's still the fastest around, at least until there is some major breakthrough in programming language design.
rbehrends 5y ago
No, not really. Languages like Nim/Rust/D/etc. should not have significantly different speeds for equivalent implementations for these kinds of benchmarks, as they ultimately can be written to compile to equivalent LLVM IR and thus the same machine code. What you're usually seeing when you see such large differences is that there are differences in the implementation of the algorithm or of a standard library function. There's nothing about these kinds of languages that should result in inherently significant performance differences on such synthetic microbenchmarks.
- qsort 5y ago
  Doesn't Nim come with a runtime, though? Rust vs. C++ is entirely due to implementation differences, but isn't this more like C++ vs. Golang?
  - rbehrends 5y ago
    A runtime isn't some sort of magic thing that inserts itself into hotspots. Several of these benchmarks are just looping over pointerfree arrays for almost all of their runtime and still show differences. Unless you call parts of the runtime, implicitly or explicitly, it shouldn't result in overhead. Allocation-heavy or write-barrier-heavy code might make a difference, but you literally have performance differences while iterating over arrays of scalars. And even then, last I benchmarked it, Nim actually compared favorably with e.g. jemalloc.
    - yakubin 5y ago
      It depends on the runtime. In the case of Go e.g. the compiler inserts potential "yield"-s at every function call[1]. And in the case of Swift the atomic "release" calls at ends of scopes will run regardless whether there are hot loops or not. So it all depends on the language. Also, you pointed to 3 languages with 3 different memory management strategies and 3 different toolchains (Rust is LLVM-based; Nim compiles to C and then calls a C compiler, GCC recommended; D has its own non-GCC non-LLVM compiler). It feels weird to put them in the same bag implementation-wise. [1]: <https://golangbyexample.com/goroutines-golang/ https://golangbyexample.com/goroutines-golang/> (potentially stale information)
boongeheYOE 5y ago
By a factor of 5 to 10 is not expected.
mratsim 5y ago
In my benchmarks, Nim is faster than Rust: - multithreading runtime (i.e Rayon vs Weave https://github.com/mratsim/weave https://github.com/mratsim/weave) - Cryptography: https://hackmd.io/@gnark/eccbench#Pairing https://hackmd.io/@gnark/eccbench#Pairing - Scientific computing / matrix multiplication: https://github.com/bluss/matrixmultiply/issues/34#issuecomment-445412450 https://github.com/bluss/matrixmultiply/issues/34#issuecomme... There is no inherent reason why a Nim program would be slower than Rust.

So Rust outperforms Nim on all benchmarks? That's kind of expected right?

by jstx1 5y ago

So Rust outperforms Nim on all benchmarks? That's kind of expected right?

I think so, but I think the performance gap is a little larger than I was expecting with Nim effectively compiling to C as an intermediary. https://programming-language-benchmarks.vercel.app/rust-vs-c https://programming-language-benchmarks.vercel.app/rust-vs-c https://programming-language-benchmarks.vercel.app/nim-vs-c https://programming-language-benchmarks.vercel.app/nim-vs-c

MisterTea 5y ago
> I was expecting with Nim effectively compiling to C as an intermediary. C has nothing to do with performance, runtimes and libraries do. Nim has runtime overhead whereas C and Rust do not.
- samwillis 5y ago
  Quite true, although my (possibly naive) assumption was that with it being a statically typed language and these being "simple" algorithmic benchmarks the gap would be closer. Not expecting it to match Rust or C in performance, just surprised at the size of the gap. Would be interesting to see what the generated C is like from the Nim code.
  - MisterTea 5y ago
    > Would be interesting to see what the generated C is like from the Nim code. Just did a few quick tests. A hello world in nim is a one liner: echo "hello world" It compiles to a 102136 byte binary on Linux. A hello world c program consisting of a single printf is 16696 bytes.
    - shirleyquirk 5y ago
      https://hookrace.net/blog/nim-binary-size/ https://hookrace.net/blog/nim-binary-size/ gets hello world to parity(17k) with `-d:release --opt:size` and `strip`, and then shows how to get that down to 150 bytes.
    - xigoi 5y ago
      Did you compile it in release mode?
- arc619 5y ago
  ...both Rust and C have runtimes: https://blog.mgattozzi.dev/rusts-runtime/ https://blog.mgattozzi.dev/rusts-runtime/
- mratsim 5y ago
  The runtime overhead is type-specific. If you don't use automatically managed types like seq/string and ref, there is no GC underneath. And even then, the GC is just reference counting so no difference from Rust Rc/Arc.
cogman10 5y ago
C is an easy intermediary, C is a bad intermediary. The issue with targeting C is you ended up losing a lot of information about intent. Compilers looking at C have to put a lot of trust that what the programmer wrote was intentional (and thus, unoptimizable). Rust and C both end up faster than Nim because they are talking directly to the compiler. IDK enough about Nim to say if it could hit Rust/C speeds if it targeted the LLVM instead, but it would be a lot faster than it currently is targeting C.
- masklinn 5y ago
  > C is an easy intermediary, C is a bad intermediary. > The issue with targeting C is you ended up losing a lot of information about intent. Compilers looking at C have to put a lot of trust that what the programmer wrote was intentional (and thus, unoptimizable). The same issue exists in C-sourced compiler backends, which is both GCC and LLVM. It’s not exactly a secret that anything which is not strongly exercised by standard C(++) codebases tends to be rather broken for any non-trivial use and takes years to fix (restrict/noalias being the poster child for this, at least in the context of Rust, I think it hasn’t gotten re-disabled yet so it might make a year).
  - cogman10 5y ago
    I think that has less to do with LLVM/GCC being C(++) implementations and more to do with the fact that the primary language compiled by these compilers is C and C++. Other languages that target these compilers aren't used nearly to the same level (maybe swift now?). The JVM, for example, doesn't see the same issue so much with non-Java languages. Mainly because (IMO) it's been hosting non-java languages for several years now.

deepsun 5y ago

It's much better to start improving from the lowest point :)

capableweb 5y ago

> That's kind of expected right? I'm not sure. The following claim is being made on Nim's website: > Modern concepts like zero-overhead iterators and compile-time evaluation of user-defined functions, in combination with the preference of value-based datatypes allocated on the stack, lead to extremely performant code. > Efficient, expressive, elegant > Nim is a statically typed compiled systems programming language A systems programming language need to have exceptionally good performance, so not sure it's to be expected that the performance is so different than Rust. Apparently the Rust versions have received more optimization efforts compared to the Nim code as well, and it seems some people are working on optimizing the Nim versions so we'll see where it lands in the future I guess: https://news.ycombinator.com/item?id=30244188 https://news.ycombinator.com/item?id=30244188

carlmr 5y ago
>Apparently the Rust versions have received more optimization efforts compared to the Nim code as well Having programmed in both I think Rust often makes it easier to write fast code by making all the copying vs borrowing more explicit. Nim is obviously still very fast and much easier to pick up, so there are trade-offs here.
gameswithgo 5y ago
I've been programming for like 30 years and I've seen dozens of language show up claiming "C like performance" and so far Rust is the only one where that has been true.
- capableweb 5y ago
  Indeed, probably the most overused selling pitch for programming languages has to be "C like performance", while only providing evidence for a few smaller use cases. I've only dabbled with Rust enough to grok the ownership parts, but I'm happy to hear you feel it lives up to that. As an aside, I'm not expecting a lot of languages to live up to the "C like performance" claim when they make it, as C has been receiving optimizations for over 50 years by now, so it makes sense it's still the fastest around, at least until there is some major breakthrough in programming language design.

rbehrends 5y ago

No, not really. Languages like Nim/Rust/D/etc. should not have significantly different speeds for equivalent implementations for these kinds of benchmarks, as they ultimately can be written to compile to equivalent LLVM IR and thus the same machine code. What you're usually seeing when you see such large differences is that there are differences in the implementation of the algorithm or of a standard library function. There's nothing about these kinds of languages that should result in inherently significant performance differences on such synthetic microbenchmarks.

qsort 5y ago
Doesn't Nim come with a runtime, though? Rust vs. C++ is entirely due to implementation differences, but isn't this more like C++ vs. Golang?
- rbehrends 5y ago
  A runtime isn't some sort of magic thing that inserts itself into hotspots. Several of these benchmarks are just looping over pointerfree arrays for almost all of their runtime and still show differences. Unless you call parts of the runtime, implicitly or explicitly, it shouldn't result in overhead. Allocation-heavy or write-barrier-heavy code might make a difference, but you literally have performance differences while iterating over arrays of scalars. And even then, last I benchmarked it, Nim actually compared favorably with e.g. jemalloc.
  - yakubin 5y ago
    It depends on the runtime. In the case of Go e.g. the compiler inserts potential "yield"-s at every function call[1]. And in the case of Swift the atomic "release" calls at ends of scopes will run regardless whether there are hot loops or not. So it all depends on the language. Also, you pointed to 3 languages with 3 different memory management strategies and 3 different toolchains (Rust is LLVM-based; Nim compiles to C and then calls a C compiler, GCC recommended; D has its own non-GCC non-LLVM compiler). It feels weird to put them in the same bag implementation-wise. [1]: <https://golangbyexample.com/goroutines-golang/ https://golangbyexample.com/goroutines-golang/> (potentially stale information)

boongeheYOE 5y ago

By a factor of 5 to 10 is not expected.

mratsim 5y ago

In my benchmarks, Nim is faster than Rust: - multithreading runtime (i.e Rayon vs Weave https://github.com/mratsim/weave https://github.com/mratsim/weave) - Cryptography: https://hackmd.io/@gnark/eccbench#Pairing https://hackmd.io/@gnark/eccbench#Pairing - Scientific computing / matrix multiplication: https://github.com/bluss/matrixmultiply/issues/34#issuecomment-445412450 https://github.com/bluss/matrixmultiply/issues/34#issuecomme... There is no inherent reason why a Nim program would be slower than Rust.