6 ms·

These benchmarks have many flaws. Consider this thread: https://www.reddit.com/r/golang/comments/51mhzv/on_the_binarytrees_benchmark https://www.reddit.com/r/g

These benchmarks have many flaws. Consider this thread: https://www.reddit.com/r/golang/comments/51mhzv/on_the_binarytrees_benchmark https://www.reddit.com/r/golang/comments/51mhzv/on_the_binar... The binary-trees benchmark would fare significantly better in Go if it were allowed to use an arena allocator like other implementations. But its not. This despite the fact that the Rust version literally uses one: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/binarytrees-rust-2.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/.... I guess because its not in the stdlib in Go? Course its not in C either, so does C use malloc? Nope. It uses a memory pool too: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/binarytrees-gcc-3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Its sad that these benchmarks get used for anything. They tell you next to nothing significant.

dvfjsdhgfv 8y ago
> I guess because its not in the stdlib in Go? It's a game, and the sources are available. Nobody is stopping Golang enthusiasts from posting the results with a slab allocator or some other third-party packages.
- cdoxsey 8y ago
  Yes the maintainer is. People have tried.
  - dvfjsdhgfv 8y ago
    In that case you can set up a separate site with the results. They will be of more value since they'll be more complete.
  - igouy 8y ago
    Which popular third-party Go slab-allocator or arena allocator was used?
- masklinn 8y ago
  > Nobody is stopping Golang enthusiasts from posting the results with a slab allocator or some other third-party packages. You've never witnessed the "game"'s maintainer interacting with people have you?
  - MikkoFinell 8y ago
    What is it like?
    - igouy 8y ago
      For example, https://salsa.debian.org/benchmarksgame-team/benchmarksgame/issues/5#note_23886 https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
glangdale 8y ago
There are many predictable things in life: the tendency of up-and-comer languages to blitz these benchmarks due to careful optimization, and the tendency of people whose pet language has lost on a benchmark or two to get on here and write off the idea that anyone should ever benchmark cross-language. Some of the benchmarks are pretty ridiculous and flawed. Others look pretty straightforwardly comparable. I have trouble looking at n-body (just fr'instance) and seeing any major difference or cheat on the Rust side (or for that matter, the much more mature C/C++ sides). Leaving aside the obviously ridiculous (the regex one is a absolute festival of 'who brought a fast regex library to the gunfight'), the trend is pretty clear: Go's code generator is pretty ordinary (it's not actually aimed at achieving ultimate code quality at any compile-time cost, so it's working as designed) and GC isn't free.
- cdoxsey 8y ago
  I find microbenchmarks mostly useless. There are just too many variables in play. Also they tend to focus on the developer who optimizes code, not the journeyman developer in your company who actually writes code. C or Fortran looks fantastic on benchmarks, but if you forced all your developers to use C I doubt you'd see a similar outcome. They seem to push developers in the direction of coming up with simple rules of thumb like "Go is 2x slower than C" or "Python is 10x slower than C" which is complete nonsense. It really matters what the code is doing. In particular for server programs developers come to these conclusions without considering the multi-core environment they're working in. ie, redis is a really fast key-value store that will use 1 core on a 32 core machine. And as it turns out all the tips and tricks to improving performance in these cases works in all the other languages too. So just what are we to conclude from them? I suppose a nice simple math benchmark can demonstrate that LLVM or GCC is better at optimizing code than Go?
  - glangdale 8y ago
    I have a lot of sympathy for your viewpoint, as I think it's abundantly clear that there are major productivity differences between languages. I doubt that anyone would be seriously running around doing an end-zone dance because C beats, say, Python. "Yeah, suck that, Python losers!". However, I think the fairly simple benchmarks where there looks to be a rough 1:1 correspondence between the structures of some similar Rust, Fortran, C/C++, Go code might demonstrate exactly the point you suggested (facetiously? I'm not quite sure how you are suggesting we compare compiler frameworks to a language, especially a language that can use both frameworks with alternate backends...) I don't know why this is controversial. The Go optimizer wasn't written primarily with utmost code quality in mind (it's pretty much a lift of the Plan 9 C compiler, which I worked with in 1994). It's designed to produce code quickly. It's also not rocket science that GC can be expensive despite years of claims to the contrary (it's been 'about to be the same or faster as manual memory management' since the mid-90s at least). Leaving aside the wacky benchmarks (binary trees and regex) the results seem pretty consistent with this. These rules of thumb are OK for what they are worth (not very much). Eventually we do have to think about how much performance can be extracted from a language if we really need it. The irritating thing about this whole benchmarking problem is that there is no natural population of programs to draw from, and there is constant thumb-on-the-scale cheating (who can intrinsic the hardest for Grandpa's ancient Core 2 duo, but no later?) - agreed. Essentially we're seeing a real-life version of the saying "the plural of anecdote is not 'data'". Nonetheless... it feels like we can say some programming languages are faster than others.
kevin_b_er 8y ago
So, we're basically looking at the difference between malloc and an arena allocator?
riku_iki 8y ago
> I guess because its not in the stdlib in Go? I think the premise of these specific benchmark is to use language idiomatic approaches instead of hacky optimizations. Is arena allocator idiomatic for go? What library people usually use? I tried to do google search, and found this one, but it is last updated 3 years ago.. https://github.com/couchbase/go-slab https://github.com/couchbase/go-slab
steveklabnik 8y ago
A while back, there was some stuff about Rust and HashMaps and things. It ended up with this comment: https://www.reddit.com/r/rust/comments/5rwwrv/chashmap_efficient_concurrent_hash_maps_in_rust/ddifssa/?context=5 https://www.reddit.com/r/rust/comments/5rwwrv/chashmap_effic... I don't know if that policy has changed in the last year, but last I heard, that's the rules regarding this. My understanding is that the intention is to use what's usual in each language.
igouy 8y ago
>> Go if it were allowed to use an arena allocator Go is allowed to use an arena allocator. >> I guess because its not in the stdlib in Go? A popular third-party library implementation would be fine. For example, couchbase/go-slab except that it doesn't seem to be a live project and there's no indication that it is actually used by the Go community. >> so does C use malloc? C using malloc -- https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/binarytrees-gcc-1.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

dvfjsdhgfv 8y ago

> I guess because its not in the stdlib in Go? It's a game, and the sources are available. Nobody is stopping Golang enthusiasts from posting the results with a slab allocator or some other third-party packages.

cdoxsey 8y ago
Yes the maintainer is. People have tried.
- dvfjsdhgfv 8y ago
  In that case you can set up a separate site with the results. They will be of more value since they'll be more complete.
- igouy 8y ago
  Which popular third-party Go slab-allocator or arena allocator was used?
masklinn 8y ago
> Nobody is stopping Golang enthusiasts from posting the results with a slab allocator or some other third-party packages. You've never witnessed the "game"'s maintainer interacting with people have you?
- MikkoFinell 8y ago
  What is it like?
  - igouy 8y ago
    For example, https://salsa.debian.org/benchmarksgame-team/benchmarksgame/issues/5#note_23886 https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...

glangdale 8y ago

There are many predictable things in life: the tendency of up-and-comer languages to blitz these benchmarks due to careful optimization, and the tendency of people whose pet language has lost on a benchmark or two to get on here and write off the idea that anyone should ever benchmark cross-language. Some of the benchmarks are pretty ridiculous and flawed. Others look pretty straightforwardly comparable. I have trouble looking at n-body (just fr'instance) and seeing any major difference or cheat on the Rust side (or for that matter, the much more mature C/C++ sides). Leaving aside the obviously ridiculous (the regex one is a absolute festival of 'who brought a fast regex library to the gunfight'), the trend is pretty clear: Go's code generator is pretty ordinary (it's not actually aimed at achieving ultimate code quality at any compile-time cost, so it's working as designed) and GC isn't free.

cdoxsey 8y ago
I find microbenchmarks mostly useless. There are just too many variables in play. Also they tend to focus on the developer who optimizes code, not the journeyman developer in your company who actually writes code. C or Fortran looks fantastic on benchmarks, but if you forced all your developers to use C I doubt you'd see a similar outcome. They seem to push developers in the direction of coming up with simple rules of thumb like "Go is 2x slower than C" or "Python is 10x slower than C" which is complete nonsense. It really matters what the code is doing. In particular for server programs developers come to these conclusions without considering the multi-core environment they're working in. ie, redis is a really fast key-value store that will use 1 core on a 32 core machine. And as it turns out all the tips and tricks to improving performance in these cases works in all the other languages too. So just what are we to conclude from them? I suppose a nice simple math benchmark can demonstrate that LLVM or GCC is better at optimizing code than Go?
- glangdale 8y ago
  I have a lot of sympathy for your viewpoint, as I think it's abundantly clear that there are major productivity differences between languages. I doubt that anyone would be seriously running around doing an end-zone dance because C beats, say, Python. "Yeah, suck that, Python losers!". However, I think the fairly simple benchmarks where there looks to be a rough 1:1 correspondence between the structures of some similar Rust, Fortran, C/C++, Go code might demonstrate exactly the point you suggested (facetiously? I'm not quite sure how you are suggesting we compare compiler frameworks to a language, especially a language that can use both frameworks with alternate backends...) I don't know why this is controversial. The Go optimizer wasn't written primarily with utmost code quality in mind (it's pretty much a lift of the Plan 9 C compiler, which I worked with in 1994). It's designed to produce code quickly. It's also not rocket science that GC can be expensive despite years of claims to the contrary (it's been 'about to be the same or faster as manual memory management' since the mid-90s at least). Leaving aside the wacky benchmarks (binary trees and regex) the results seem pretty consistent with this. These rules of thumb are OK for what they are worth (not very much). Eventually we do have to think about how much performance can be extracted from a language if we really need it. The irritating thing about this whole benchmarking problem is that there is no natural population of programs to draw from, and there is constant thumb-on-the-scale cheating (who can intrinsic the hardest for Grandpa's ancient Core 2 duo, but no later?) - agreed. Essentially we're seeing a real-life version of the saying "the plural of anecdote is not 'data'". Nonetheless... it feels like we can say some programming languages are faster than others.

kevin_b_er 8y ago

So, we're basically looking at the difference between malloc and an arena allocator?

riku_iki 8y ago

> I guess because its not in the stdlib in Go? I think the premise of these specific benchmark is to use language idiomatic approaches instead of hacky optimizations. Is arena allocator idiomatic for go? What library people usually use? I tried to do google search, and found this one, but it is last updated 3 years ago.. https://github.com/couchbase/go-slab https://github.com/couchbase/go-slab

steveklabnik 8y ago

A while back, there was some stuff about Rust and HashMaps and things. It ended up with this comment: https://www.reddit.com/r/rust/comments/5rwwrv/chashmap_efficient_concurrent_hash_maps_in_rust/ddifssa/?context=5 https://www.reddit.com/r/rust/comments/5rwwrv/chashmap_effic... I don't know if that policy has changed in the last year, but last I heard, that's the rules regarding this. My understanding is that the intention is to use what's usual in each language.

igouy 8y ago

>> Go if it were allowed to use an arena allocator Go is allowed to use an arena allocator. >> I guess because its not in the stdlib in Go? A popular third-party library implementation would be fine. For example, couchbase/go-slab except that it doesn't seem to be a live project and there's no indication that it is actually used by the Go community. >> so does C use malloc? C using malloc -- https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/binarytrees-gcc-1.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...