7 ms·

Rust Faster

114 points by jaxondu 11y ago

geofft 11y ago
Fascinating things I learned from this article: 1. The benchmarks game is mostly a game of who can solve the problem fastest with the constraint of "all the code has to be in this language." It's not about how anyone would realistically write the code; if ludicrous optimizations were in-scope for any real-world project, so would be calling out to a different language. It's worse than the usual problem of benchmarks being unrepresentative of real code; the implementations are also unrepresentative. 2. Apparently performant Haskell involves unchecked raw memory access? What's the story there?
- jacobolus 11y ago
  All of these cross-language benchmarks inevitably end up like this. They tend to tell you as much or more about the cleverness/knowledge/available free time of the people writing the specific implementation in each language as they tell you about the languages’ inherent differences. I don’t think it’s a solvable problem though, as there’s no perfect definition of “how anyone would realistically write the code”. Realistically, people are going to write the first naïve implementation they can think of until it becomes a performance problem and someone has time/resources to throw at speeding things up. In the case that a piece of code is a bottleneck and performance matters enough to devote engineering effort, folks will start doing all kinds of crazy “unrealistic” optimizations to real-world code.
  - fauigerzigerk 11y ago
    I think the benchmarks would tell us much more if they compared implementations of text book algorithms (and data structures) instead of comparing solutions to text book problems. I realize that this would not necessarily make the code any more realistic or idiomatic. On the contrary, it could make the code extremely contrived for some algorithms in some languages. Some combinations might even be unimplementable. But I don't care about that, because, as you say, what is and isn't idiomatic is very subjective and it gets thrown out anyway once we decide that some piece of code does need to run fast. What makes these benchmarks pretty useless is that you don't know whether the choice of algorithm or the choice of programming language dominates a particular result.
    - titanomachy 11y ago
      Well, you can learn something if you read the code at least.
    - igouy 11y ago
      >>implementations of text book algorithms<< You can do that comparison and publish the results. >> choice of algorithm or the choice of programming language<< Do the n-body programs implement different algorithms? http://benchmarksgame.alioth.debian.org/u64q/performance.php?test=nbody http://benchmarksgame.alioth.debian.org/u64q/performance.php...
      - fauigerzigerk 11y ago
        Sorry, my mistake. I thought that they did not mandate one particular algorithm, but apparently I was wrong. They say: "We are trying to show the performance of various programming language implementations - so we ask that contributed programs not only give the correct result, but also use the same algorithm to calculate that result." Thanks for the correction.
        igouy 11y ago
        Well, "we ask" is not the same as "we receive" :-) Also, because of the differences between such a wide range of programming languages, there has to be some slop and flexibility in the way the programs are allowed to be written -- otherwise it really would be C written in Haskell. http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.html#comparison http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
    - deleted 11y ago
      [deleted]
- kbenson 11y ago
  Well, yes. Another way to look at the benchmark game is "if I'm not using a fast low-level language, theoretically what speed gain might I expect if I wrote a portion of my project in X language and called out to it (ignoring data marshaling)?" It would be interesting if entries were categorized into idiomatic and non-idiomatic representations of that language, but that would also require people agree on what is idiomatic for that language, which is easier said than done in a lot of cases.
- jamii 11y ago
  If you just look at the 'best' numbers for each language and treat it as a fight, it's pretty useless. But if you are evaluating a new language it can be very educational to look at all the solutions for that language and see the range of tradeoffs between speed, space and readability. The short and simple programs show you what kind of performance you can expect from straightforward, idiomatic code and the crazy optimised versions show you how far you can push the performance bracket and what you have to sacrifice to get there (performance, safety etc).
  - igouy 11y ago
    If we just look at the 'best' numbers for each language and treat it as a fight -- we've chosen to do something pretty useless ;-)
- deleted 11y ago
  [deleted]
- igouy 11y ago
  >>ludicrous optimizations<< If only you'd look at the code. There can be several quite different programs for the same language implementation: http://benchmarksgame.alioth.debian.org/u64q/program.php?test=spectralnorm&lang=gpp&id=1 http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...
- qznc 11y ago
  > the constraint of "all the code has to be in this language." No that is not a usual constraint, many implementations use third party libraries. E.g., regexdna implementations usually rely on some regex library and pidigits just measures libgmp.
  - steveklabnik 11y ago
    The "third party" bit is hard to define as well. For example, Rust doesn't include regular expressions in the standard library, because we have made an explicit choice to keep the standard library small. But the regex package is maintained by the Rust team, so it's not _exactly_ third party. But some languages with regexes in the language just have bindings to PCRE or something similar... so it's "first party" in the sense that it's built-in, but "third party" in the sense that the code wasn't really written and isn't maintained by them.
  - mcguire 11y ago
    It is a valid constraint, though. Many long-standing complaints from the Java side of things were along the lines of "We can't use libgmp."
    - igouy 11y ago
      http://benchmarksgame.alioth.debian.org/u64q/program.php?test=pidigits&lang=java&id=2#sourcecode http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...
dikaiosune 11y ago
Very cool, and definitely gives me some insight into those benchmarks. Which makes me wonder -- are there benchmarks for "boring" programs in a variety of languages? I'm generally more interested in the execution speed of implementations that I would actually have time to write when on a deadline.
- kibwen 11y ago
  The best thing for your purposes would be a site that attempted to optimize for the appearance of idiomatic code, which you could then secretly use to gauge the performance of code that's idiomatic to each language. In other words, entirely remove performance showboating as an incentive. Rosetta Code (http://rosettacode.org/wiki/Rosetta_Code http://rosettacode.org/wiki/Rosetta_Code) may be the closest to what you want, though I can guess that most of the Rust implementations were written prior to 1.0 and won't even compile anymore, let alone represent idiomatic code. :P
  - dikaiosune 11y ago
    I like that idea -- although I would agree that Rosetta Code often does not represent idiomatic code, even for longer-established languages. The wikibooks algorithms book might also be a good candidate, although I've not yet seen Rust on there.
- igouy 11y ago
  If only you'd look at the code, you might get some insight. You might see some boring programs. You could always contribute some boring programs ;-)
- gus_massa 11y ago
  Many languages have a few programs for each benchmark. You can try looking at the shortest one. Most submitters try to optimize for speed, and to use smart tricks you need more LOC, so usually the shortest program is the more straightforward. Actually, in this site they don't measure the size of a program in LOC, they use the size of the compressed program that is a better metric. (It may be different in other sites specialized in code golf, because in the those sites the submitters are specializing for size specifically.)
techdragon 11y ago
Every time I look at rust and posts about rust like this one. It occurs to me that "I can use this to write 'C' libraries" and a variable number of moments later, I think to myself that I'll keep Go at the bottom of my list of languages to learn to program in, after Ruby, Lisp, Fortran, COBOL, and Intercal, I'll get to Go one day, if it's still relevant to me, for now Rust + Erlang / Elixir on FreeBSD is like having my own personal unicorn. I've been meaning to work on a proper "Erlang/OTP-ish" framework for Python for a long time, Pulsar[1] is a good start but needs more developers, and more documentation in order to grow. It has an example that does web sockets with normal Django, no massive hacks. (Which someone more familiar with Django should really check out and talk about more widely) But no Flask example? 1 - https://github.com/quantmind/pulsar https://github.com/quantmind/pulsar
- mcguire 11y ago
  "a proper "Erlang/OTP-ish" framework" Rust's original M:N threading almost begged for an OTP-style approach (as in, it looked like Erlang with a non-Prolog syntax :-)). But I think the current threading interfaces are pretty welcoming for an OTP style. Anyone working on that?
Everlag 11y ago
I've been working with rust for about a week now and I've found it a mirror of go in several ways. Go aims to be simple with the aim of being 'easy' to start writing idiomatically within a few days of jumping into the language. Comparatively, rust is a behemoth in terms of complexity. There's the completely relevant compilation speed difference: if rust could compile within 5-10x the time go does, it would be much more pleasant to work with. Being able to compile within 2 seconds on large projects is crazy for iteration speed; having to wait upwards of 10 seconds on toy projects is not. However, rust is stupid fast while also being safer than go. Also, generics; that's a flamewar for another day. Context: I've been working with go for around 2-3 years now. I recently decided to pick up rust because of the guarantees it provides along with the crazy performance.
- kibwen 11y ago
  You'll be happy to hear that compilation speed is the foremost priority of the Rust developers at the moment, and is the result of a technical-debt-laden codebase (having survived four years of traumatic language evolution) rather than any intrinsic property of the language. :) Huge improvements are in the pipe.
  - Everlag 11y ago
    Fantastic to hear! I don't even want to think about the horrors that 4 years of heavy core changes would do to a language.
    - kibwen 11y ago
      The bottom line is that self-hosting is a great way to shake down your language on a large-ish codebase and a fine way of encouraging outsider contributions, but should really only be done once you actually have a relatively solid idea of what your language is going to be. :)
  - IshKebab 11y ago
    Do you have any evidence that it isn't an intrinsic property of the language? I would have thought all that type inference is non-trivial to calculate.
imaginenore 11y ago
The requirement to read line by line is ridiculous.
fijal 11y ago
"A staple of all performance discussion is the great Computer Language Benchmarks Game." this is so sad, but true
mcguire 11y ago
"k_nucleotide is a test of how fast your Hash map is." Speaking of Rusts' HashMap, the Robin Hood map is pretty darn sweet (and I say that as someone who translated a Pythonish map to Rust), but the last time I looked it was still throttled by SipHash.[1] Is there any progress on SIMD-ing that? [1] "Safety vs. performance", etc., etc.
- pcwalton 11y ago
  SipHash as implemented in Rust is actually pretty fast for long keys; it's only for short keys that it falls down (and SIMDification won't help too much on short keys).
  - mcguire 11y ago
    What's a "long key" in this case? I've only looked at it for modest-length strings.
acconsta 11y ago
The author also points out that some of the benchmarks poorly represent real workloads: "Bottom up (since the worst offenders are now first), binary-trees is silly since it measures allocation speed for a case that simply doesn't exist in real code; thread-ring is basically insane, since nobody ever bottlenecks like that; chameneos-redux's C++ implementation is ridiculous. The C is not so ridiculous, but you still have the problem that basically every language in the top few spots does something completely different; pidigits tests whether you have bindings to GMP; regex-dna tests a regex engine on a small subset of cases (arguably the first half-acceptable benchmark); k-nucleotide tests who has the best hash table for this particular silly scheme, and they don't all even do the same thing (eg. Scala precompacts, like my new Rust version); mandelbrot is kind'a OK; reverse-complement would be kind'a OK if not for a few hacky implementations (like the Rust); spectral-norm is kind'a OK; Haskell basically cheats fasta (which is why I copied it); meteor-contest is too short to mean anything at all; fannkuch-redux is probably kind'a OK, n-body is kind'a OK. So maybe 5/13 are acceptable, and I'd still only use 4 of those. I think if looking at mandelbrot, spectral-norm, fannkuch-redux and n-body you can argue the benches are a reasonable measure of peak performance. However, these cases are also all too small and simple to really be convincing either, nor is it particularly fair (where's NumPy for Python?)." https://users.rust-lang.org/t/blog-rust-faster/3117/12?u=acconsta https://users.rust-lang.org/t/blog-rust-faster/3117/12?u=acc...
- igouy 11y ago
  >>bad at simulating real workloads<< Have you looked at the benchmarks game website? Please show where the benchmarks game website claims that those tasks simulate "real workloads" (whatever that means). You will see "Your application is the ultimate benchmark" and "These are just 10 tiny examples" and … http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.html http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
  - acconsta 11y ago
    I apologize if I gave the impression I was claiming otherwise. It's just context for the discussion.

114 points by jaxondu 11y ago

Fascinating things I learned from this article: 1. The benchmarks game is mostly a game of who can solve the problem fastest with the constraint of "all the code has to be in this language." It's not about how anyone would realistically write the code; if ludicrous optimizations were in-scope for any real-world project, so would be calling out to a different language. It's worse than the usual problem of benchmarks being unrepresentative of real code; the implementations are also unrepresentative. 2. Apparently performant Haskell involves unchecked raw memory access? What's the story there?

jacobolus 11y ago
All of these cross-language benchmarks inevitably end up like this. They tend to tell you as much or more about the cleverness/knowledge/available free time of the people writing the specific implementation in each language as they tell you about the languages’ inherent differences. I don’t think it’s a solvable problem though, as there’s no perfect definition of “how anyone would realistically write the code”. Realistically, people are going to write the first naïve implementation they can think of until it becomes a performance problem and someone has time/resources to throw at speeding things up. In the case that a piece of code is a bottleneck and performance matters enough to devote engineering effort, folks will start doing all kinds of crazy “unrealistic” optimizations to real-world code.
- fauigerzigerk 11y ago
  I think the benchmarks would tell us much more if they compared implementations of text book algorithms (and data structures) instead of comparing solutions to text book problems. I realize that this would not necessarily make the code any more realistic or idiomatic. On the contrary, it could make the code extremely contrived for some algorithms in some languages. Some combinations might even be unimplementable. But I don't care about that, because, as you say, what is and isn't idiomatic is very subjective and it gets thrown out anyway once we decide that some piece of code does need to run fast. What makes these benchmarks pretty useless is that you don't know whether the choice of algorithm or the choice of programming language dominates a particular result.
  - titanomachy 11y ago
    Well, you can learn something if you read the code at least.
  - igouy 11y ago
    >>implementations of text book algorithms<< You can do that comparison and publish the results. >> choice of algorithm or the choice of programming language<< Do the n-body programs implement different algorithms? http://benchmarksgame.alioth.debian.org/u64q/performance.php?test=nbody http://benchmarksgame.alioth.debian.org/u64q/performance.php...
    - fauigerzigerk 11y ago
      Sorry, my mistake. I thought that they did not mandate one particular algorithm, but apparently I was wrong. They say: "We are trying to show the performance of various programming language implementations - so we ask that contributed programs not only give the correct result, but also use the same algorithm to calculate that result." Thanks for the correction.
      - igouy 11y ago
        Well, "we ask" is not the same as "we receive" :-) Also, because of the differences between such a wide range of programming languages, there has to be some slop and flexibility in the way the programs are allowed to be written -- otherwise it really would be C written in Haskell. http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.html#comparison http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
  - deleted 11y ago
    [deleted]
kbenson 11y ago
Well, yes. Another way to look at the benchmark game is "if I'm not using a fast low-level language, theoretically what speed gain might I expect if I wrote a portion of my project in X language and called out to it (ignoring data marshaling)?" It would be interesting if entries were categorized into idiomatic and non-idiomatic representations of that language, but that would also require people agree on what is idiomatic for that language, which is easier said than done in a lot of cases.
jamii 11y ago
If you just look at the 'best' numbers for each language and treat it as a fight, it's pretty useless. But if you are evaluating a new language it can be very educational to look at all the solutions for that language and see the range of tradeoffs between speed, space and readability. The short and simple programs show you what kind of performance you can expect from straightforward, idiomatic code and the crazy optimised versions show you how far you can push the performance bracket and what you have to sacrifice to get there (performance, safety etc).
- igouy 11y ago
  If we just look at the 'best' numbers for each language and treat it as a fight -- we've chosen to do something pretty useless ;-)
deleted 11y ago
[deleted]
igouy 11y ago
>>ludicrous optimizations<< If only you'd look at the code. There can be several quite different programs for the same language implementation: http://benchmarksgame.alioth.debian.org/u64q/program.php?test=spectralnorm&lang=gpp&id=1 http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...
qznc 11y ago
> the constraint of "all the code has to be in this language." No that is not a usual constraint, many implementations use third party libraries. E.g., regexdna implementations usually rely on some regex library and pidigits just measures libgmp.
- steveklabnik 11y ago
  The "third party" bit is hard to define as well. For example, Rust doesn't include regular expressions in the standard library, because we have made an explicit choice to keep the standard library small. But the regex package is maintained by the Rust team, so it's not _exactly_ third party. But some languages with regexes in the language just have bindings to PCRE or something similar... so it's "first party" in the sense that it's built-in, but "third party" in the sense that the code wasn't really written and isn't maintained by them.
- mcguire 11y ago
  It is a valid constraint, though. Many long-standing complaints from the Java side of things were along the lines of "We can't use libgmp."
  - igouy 11y ago
    http://benchmarksgame.alioth.debian.org/u64q/program.php?test=pidigits&lang=java&id=2#sourcecode http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...

dikaiosune 11y ago

Very cool, and definitely gives me some insight into those benchmarks. Which makes me wonder -- are there benchmarks for "boring" programs in a variety of languages? I'm generally more interested in the execution speed of implementations that I would actually have time to write when on a deadline.

kibwen 11y ago
The best thing for your purposes would be a site that attempted to optimize for the appearance of idiomatic code, which you could then secretly use to gauge the performance of code that's idiomatic to each language. In other words, entirely remove performance showboating as an incentive. Rosetta Code (http://rosettacode.org/wiki/Rosetta_Code http://rosettacode.org/wiki/Rosetta_Code) may be the closest to what you want, though I can guess that most of the Rust implementations were written prior to 1.0 and won't even compile anymore, let alone represent idiomatic code. :P
- dikaiosune 11y ago
  I like that idea -- although I would agree that Rosetta Code often does not represent idiomatic code, even for longer-established languages. The wikibooks algorithms book might also be a good candidate, although I've not yet seen Rust on there.
igouy 11y ago
If only you'd look at the code, you might get some insight. You might see some boring programs. You could always contribute some boring programs ;-)
gus_massa 11y ago
Many languages have a few programs for each benchmark. You can try looking at the shortest one. Most submitters try to optimize for speed, and to use smart tricks you need more LOC, so usually the shortest program is the more straightforward. Actually, in this site they don't measure the size of a program in LOC, they use the size of the compressed program that is a better metric. (It may be different in other sites specialized in code golf, because in the those sites the submitters are specializing for size specifically.)

techdragon 11y ago

Every time I look at rust and posts about rust like this one. It occurs to me that "I can use this to write 'C' libraries" and a variable number of moments later, I think to myself that I'll keep Go at the bottom of my list of languages to learn to program in, after Ruby, Lisp, Fortran, COBOL, and Intercal, I'll get to Go one day, if it's still relevant to me, for now Rust + Erlang / Elixir on FreeBSD is like having my own personal unicorn. I've been meaning to work on a proper "Erlang/OTP-ish" framework for Python for a long time, Pulsar[1] is a good start but needs more developers, and more documentation in order to grow. It has an example that does web sockets with normal Django, no massive hacks. (Which someone more familiar with Django should really check out and talk about more widely) But no Flask example? 1 - https://github.com/quantmind/pulsar https://github.com/quantmind/pulsar

mcguire 11y ago
"a proper "Erlang/OTP-ish" framework" Rust's original M:N threading almost begged for an OTP-style approach (as in, it looked like Erlang with a non-Prolog syntax :-)). But I think the current threading interfaces are pretty welcoming for an OTP style. Anyone working on that?

Everlag 11y ago

I've been working with rust for about a week now and I've found it a mirror of go in several ways. Go aims to be simple with the aim of being 'easy' to start writing idiomatically within a few days of jumping into the language. Comparatively, rust is a behemoth in terms of complexity. There's the completely relevant compilation speed difference: if rust could compile within 5-10x the time go does, it would be much more pleasant to work with. Being able to compile within 2 seconds on large projects is crazy for iteration speed; having to wait upwards of 10 seconds on toy projects is not. However, rust is stupid fast while also being safer than go. Also, generics; that's a flamewar for another day. Context: I've been working with go for around 2-3 years now. I recently decided to pick up rust because of the guarantees it provides along with the crazy performance.

kibwen 11y ago
You'll be happy to hear that compilation speed is the foremost priority of the Rust developers at the moment, and is the result of a technical-debt-laden codebase (having survived four years of traumatic language evolution) rather than any intrinsic property of the language. :) Huge improvements are in the pipe.
- Everlag 11y ago
  Fantastic to hear! I don't even want to think about the horrors that 4 years of heavy core changes would do to a language.
  - kibwen 11y ago
    The bottom line is that self-hosting is a great way to shake down your language on a large-ish codebase and a fine way of encouraging outsider contributions, but should really only be done once you actually have a relatively solid idea of what your language is going to be. :)
- IshKebab 11y ago
  Do you have any evidence that it isn't an intrinsic property of the language? I would have thought all that type inference is non-trivial to calculate.

imaginenore 11y ago

The requirement to read line by line is ridiculous.

fijal 11y ago

"A staple of all performance discussion is the great Computer Language Benchmarks Game." this is so sad, but true

mcguire 11y ago

"k_nucleotide is a test of how fast your Hash map is." Speaking of Rusts' HashMap, the Robin Hood map is pretty darn sweet (and I say that as someone who translated a Pythonish map to Rust), but the last time I looked it was still throttled by SipHash.[1] Is there any progress on SIMD-ing that? [1] "Safety vs. performance", etc., etc.

pcwalton 11y ago
SipHash as implemented in Rust is actually pretty fast for long keys; it's only for short keys that it falls down (and SIMDification won't help too much on short keys).
- mcguire 11y ago
  What's a "long key" in this case? I've only looked at it for modest-length strings.

acconsta 11y ago

The author also points out that some of the benchmarks poorly represent real workloads: "Bottom up (since the worst offenders are now first), binary-trees is silly since it measures allocation speed for a case that simply doesn't exist in real code; thread-ring is basically insane, since nobody ever bottlenecks like that; chameneos-redux's C++ implementation is ridiculous. The C is not so ridiculous, but you still have the problem that basically every language in the top few spots does something completely different; pidigits tests whether you have bindings to GMP; regex-dna tests a regex engine on a small subset of cases (arguably the first half-acceptable benchmark); k-nucleotide tests who has the best hash table for this particular silly scheme, and they don't all even do the same thing (eg. Scala precompacts, like my new Rust version); mandelbrot is kind'a OK; reverse-complement would be kind'a OK if not for a few hacky implementations (like the Rust); spectral-norm is kind'a OK; Haskell basically cheats fasta (which is why I copied it); meteor-contest is too short to mean anything at all; fannkuch-redux is probably kind'a OK, n-body is kind'a OK. So maybe 5/13 are acceptable, and I'd still only use 4 of those. I think if looking at mandelbrot, spectral-norm, fannkuch-redux and n-body you can argue the benches are a reasonable measure of peak performance. However, these cases are also all too small and simple to really be convincing either, nor is it particularly fair (where's NumPy for Python?)." https://users.rust-lang.org/t/blog-rust-faster/3117/12?u=acconsta https://users.rust-lang.org/t/blog-rust-faster/3117/12?u=acc...

igouy 11y ago
>>bad at simulating real workloads<< Have you looked at the benchmarks game website? Please show where the benchmarks game website claims that those tasks simulate "real workloads" (whatever that means). You will see "Your application is the ultimate benchmark" and "These are just 10 tiny examples" and … http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.html http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
- acconsta 11y ago
  I apologize if I gave the impression I was claiming otherwise. It's just context for the discussion.