9 ms·

Go 1.1beta1 now on the benchmarks game

30 points by jondot 13y ago

melling 13y ago
It would be nice to compare it to the old version. At any rate, it seems like it's a lot more competitive with Java in performance and it uses a lot less memory: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=go&lang2=java&data=u64 http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te... It's certainly much faster than the fastest "scripting" language: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=go&lang2=perl&data=u64 http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...
- igouy 13y ago
  >>It would be nice to compare it to the old version.<< 12 days ago https://news.ycombinator.com/item?id=5494113 https://news.ycombinator.com/item?id=5494113
vardump 13y ago
I'd love to see how Go 1.1 compares against modern JITted scripting languages, because I think those will be a major part of Go's current and future competition. Hopefully the benchmarks game will one day include those JIT variants. V8 is there, but sadly for example PyPy, HHVM (Facebook's PHP TJIT) and LuaJIT are not.
- igouy 13y ago
  "If you're interested in something not shown then please take the program source code and the measurement scripts and publish your own measurements." http://benchmarksgame.alioth.debian.org/play.php#languagex http://benchmarksgame.alioth.debian.org/play.php#languagex
  - dilap 13y ago
    Yeah, but the request isn't to include some random language, it's to use the fast implementations of any given language. Why wouldn't you, in a site that benchmarks speed? I think not to do so is unhelpful, actually -- if you don't realize they're are JITs for Python and Lua that aren't being included in the benchmarks, you will come away with a completely wrong impression about the possible performance of those languages. edit: what spion said in their comment
    - igouy 13y ago
      >>Why would you not do that...<< "Because I want to do fewer chores not more!" >>a completely wrong impression<< "These are not the only compilers and interpreters. These are not the only programs that could be written. These are not the only tasks that could be solved. These are just 10 tiny examples."
      - dilap 13y ago
        Let's say I make a website called the Prettiest Person Game, and I throw up some professional crazy-awesome shots of myself and my buddies, and I also put up all igouy's old driver's license photos, and then create a whole bunch of different ways we can compare these photos and see who's prettier. And let's say this site becomes really popular. And people start posting articles like, "wow, dilap is like 10x prettier than igouy in ever single way!" And then igouy's friends go, "whoaaaa, hold on a minute, those are terrible photo's of ioguy! he can really look much better than that!" And then I'm like, "Hey, I ain't got time for this -- I told you already these are not the only photos in the world. Go take your own photos if you want a different comparison!" That's what you're saying.
        igouy 13y ago
        What I'm saying is -- tl;dr Measurement is highly specific -- the time taken for this benchmark task, by this program, with this programming language implementation, with these options, on this computer, with these workloads. http://benchmarksgame.alioth.debian.org/ http://benchmarksgame.alioth.debian.org/
- spion 13y ago
  I think we're in dire need of a new benchmarks game. The author: removed PyPy removed the clear overview table that showed all languages, keeping only the barely visible gray bar charts with vertical text (why not horizontal bar charts, they can be scrolled and read without head tilting) refuses to add LuaJIT and other languages responds to most change inquiries with "publish your own version" which probably means that he doesn't really care about the project anymore As a result, the spirit of the benchmarks game was lost a long time ago as its made less fun with each iteration.
  - igouy 13y ago
    Not only that, but "the author" has for many years provided the source code for the scripts to make the measurements -- so anyone who actually wants to can easily start a new benchmarks game. Go do! http://benchmarksgame.alioth.debian.org/play.php#languagex http://benchmarksgame.alioth.debian.org/play.php#languagex
    - gnuvince 13y ago
      Why ask people to start their own benchmarks (presumably with Blackjack and hookers) instead of allowing people to contribute to the Benchmark Games?
      - igouy 13y ago
        Because they don't wish to contribute to the benchmarks game, they wish to see something different -- and that's fine with me.
    - spion 13y ago
      Its far from "easily". Its not like you could just fork the code and run the benchmarks game - you would not only have to setup the environment for all languages, but also understand the not-so-well documented idiosyncrasies and deficiencies of the implementation (hardcoded paths? seriously?) It doesn't seem like the code was originally written to be understood and modified by others and its not fully documented. As such your claim that its simple to fork it isn't really valid. (However I'm willing to admit the possibility that the configuration complexity is not easily avoidable) I agree that its far from easy to write a benchmarks game, setup a test environment and invest oh-so-many CPU cycles into running the tests periodically. But I wouldn't go around telling people that forking the original is easy either - that would be very misleading. Much will have to be done to improve the current engine to make it easier to fork. I'm seriously considering doing that (or alternatively making a new benchmarks game from scratch with the intent of making it easy to configure and fork)
- rdtsc 13y ago
  That was there. PyPy was there at least. LuaJIT too I think. All go yanked out. The benchmarks are bogus anyway unless what you plan on working on n-body or fasta search or a short other list of things they run.
- fabriceleal 13y ago
  I believe Go is compiled [1][2]. Isn't a comparison with something like Haskell [3] more fair? [1] http://golang.org/doc/faq#Implementation http://golang.org/doc/faq#Implementation [2] http://golang.org/cmd/go/#hdr-Compile_and_run_Go_program http://golang.org/cmd/go/#hdr-Compile_and_run_Go_program [3] http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=go&lang2=ghc&data=u64 http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...
  - elliotlai 13y ago
    The Glorious Glasgow Haskell Compilation System is truly glorious
sudhirj 13y ago
Why the spectacular loss to V8 in the Regex tests? Is Go's regex engine a naive implementation as of now?
- chaosphere2112 13y ago
  Russ Cox, in response to a regex benchmark vs. Python/Ruby. "You assume the benchmark is worth something. First of all, Ruby and Python are using C implementations of the regexp search, so Go is being beat by C, not by Ruby. Second, Go is using a different algorithm for regexp matching than the C implementations in those other languages. The algorithm Go uses guarantees to complete in time that is linear in the length of the input. The algorithm that Ruby/Python/etc are using can take time exponential in the length of the input, although on trivial cases it typically runs quite fast."[1] [1] https://groups.google.com/d/msg/golang-nuts/6d1maef4jQ8/yg4NY5qRtsMJ https://groups.google.com/d/msg/golang-nuts/6d1maef4jQ8/yg4N...
  - baq 13y ago
    As a member of the masses of programmers that typically deal with truckloads of trivial problems, his reply is worth as much as the benchmark he criticizes. I've never used anything but a trivial regexp (for some values of trivial) in production code (one-shot problems don't count) and I don't give a damn what is being benchmarked here - I only care if my Python script runs fast.
    - ja30278 13y ago
      In this case, you really should care. http://www.benfrederickson.com/2013/04/09/python-catastrophic-regular-expressions-and-the-gil.html http://www.benfrederickson.com/2013/04/09/python-catastrophi...
    - kibwen 13y ago
      Actually, I might agree with Go here. I agree that most of my regexes are the trivial kind that don't exhibit exponential growth, but that's only because I also control the input to all those regexes. But Go's intended niche appears to be for user-facing server-side programs, and in that environment if you ever run a naive exponential-time regex against user-supplied input you're setting yourself up for a potential DOS attack.
cnlwsu 13y ago
Isn't comparing against a scripting language apples and oranges? Compiled vs interpreted one would assume to always look like this. Although the modern sophisticated JITs have definitely narrowed the playing field.
- igouy 13y ago
  http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.php#comparison http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
zobzu 13y ago
I dunno about you but I don't use Go where I could use javascript. And also, I use multicore machines only. So I prefer this URL: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=go&lang2=gcc&data=u64q http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t... :P
- tptacek 13y ago
  Go vs. v8 is a reasonable comparison. Node.js is probably the closest substitute for the way Go is most often used today (as API servers or backend app components for web applications). I have no comment as to whether this "benchmarks game" stuff tells us anything meaningful.
- burntsushi 13y ago
  ... That's using Go 1.0.3.
  - mseepgood 13y ago
    Why are some benchmarks using Go 1.0.3 and some 1.1beta1? And why is the PCRE program broken? http://benchmarksgame.alioth.debian.org/u64/program.php?test=regexdna&lang=go&id=7 http://benchmarksgame.alioth.debian.org/u64/program.php?test... As far as I understand it just needs a "go get github.com/glenn-brown/golang-pkg-pcre/src/pkg/pcre"
    - igouy 13y ago
      https://news.ycombinator.com/item?id=5565769 https://news.ycombinator.com/item?id=5565769
      - mseepgood 13y ago
        Thanks.
- xwei 13y ago
  you prefer to do multi-thread programming in C? serious?
alecbenzer 13y ago
I'd be interested in seeing compile times. In a talk posted recently Rob Pike seemed to be kind of emphasizing #include inefficiencies and bytes read by compiler to bytes of source input ratios for C/C++, and how was better at this.

30 points by jondot 13y ago

It would be nice to compare it to the old version. At any rate, it seems like it's a lot more competitive with Java in performance and it uses a lot less memory: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=go&lang2=java&data=u64 http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te... It's certainly much faster than the fastest "scripting" language: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=go&lang2=perl&data=u64 http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...

igouy 13y ago
>>It would be nice to compare it to the old version.<< 12 days ago https://news.ycombinator.com/item?id=5494113 https://news.ycombinator.com/item?id=5494113

vardump 13y ago

I'd love to see how Go 1.1 compares against modern JITted scripting languages, because I think those will be a major part of Go's current and future competition. Hopefully the benchmarks game will one day include those JIT variants. V8 is there, but sadly for example PyPy, HHVM (Facebook's PHP TJIT) and LuaJIT are not.

igouy 13y ago
"If you're interested in something not shown then please take the program source code and the measurement scripts and publish your own measurements." http://benchmarksgame.alioth.debian.org/play.php#languagex http://benchmarksgame.alioth.debian.org/play.php#languagex
- dilap 13y ago
  Yeah, but the request isn't to include some random language, it's to use the fast implementations of any given language. Why wouldn't you, in a site that benchmarks speed? I think not to do so is unhelpful, actually -- if you don't realize they're are JITs for Python and Lua that aren't being included in the benchmarks, you will come away with a completely wrong impression about the possible performance of those languages. edit: what spion said in their comment
  - igouy 13y ago
    >>Why would you not do that...<< "Because I want to do fewer chores not more!" >>a completely wrong impression<< "These are not the only compilers and interpreters. These are not the only programs that could be written. These are not the only tasks that could be solved. These are just 10 tiny examples."
    - dilap 13y ago
      Let's say I make a website called the Prettiest Person Game, and I throw up some professional crazy-awesome shots of myself and my buddies, and I also put up all igouy's old driver's license photos, and then create a whole bunch of different ways we can compare these photos and see who's prettier. And let's say this site becomes really popular. And people start posting articles like, "wow, dilap is like 10x prettier than igouy in ever single way!" And then igouy's friends go, "whoaaaa, hold on a minute, those are terrible photo's of ioguy! he can really look much better than that!" And then I'm like, "Hey, I ain't got time for this -- I told you already these are not the only photos in the world. Go take your own photos if you want a different comparison!" That's what you're saying.
      - igouy 13y ago
        What I'm saying is -- tl;dr Measurement is highly specific -- the time taken for this benchmark task, by this program, with this programming language implementation, with these options, on this computer, with these workloads. http://benchmarksgame.alioth.debian.org/ http://benchmarksgame.alioth.debian.org/
spion 13y ago
I think we're in dire need of a new benchmarks game. The author: removed PyPy removed the clear overview table that showed all languages, keeping only the barely visible gray bar charts with vertical text (why not horizontal bar charts, they can be scrolled and read without head tilting) refuses to add LuaJIT and other languages responds to most change inquiries with "publish your own version" which probably means that he doesn't really care about the project anymore As a result, the spirit of the benchmarks game was lost a long time ago as its made less fun with each iteration.
- igouy 13y ago
  Not only that, but "the author" has for many years provided the source code for the scripts to make the measurements -- so anyone who actually wants to can easily start a new benchmarks game. Go do! http://benchmarksgame.alioth.debian.org/play.php#languagex http://benchmarksgame.alioth.debian.org/play.php#languagex
  - gnuvince 13y ago
    Why ask people to start their own benchmarks (presumably with Blackjack and hookers) instead of allowing people to contribute to the Benchmark Games?
    - igouy 13y ago
      Because they don't wish to contribute to the benchmarks game, they wish to see something different -- and that's fine with me.
  - spion 13y ago
    Its far from "easily". Its not like you could just fork the code and run the benchmarks game - you would not only have to setup the environment for all languages, but also understand the not-so-well documented idiosyncrasies and deficiencies of the implementation (hardcoded paths? seriously?) It doesn't seem like the code was originally written to be understood and modified by others and its not fully documented. As such your claim that its simple to fork it isn't really valid. (However I'm willing to admit the possibility that the configuration complexity is not easily avoidable) I agree that its far from easy to write a benchmarks game, setup a test environment and invest oh-so-many CPU cycles into running the tests periodically. But I wouldn't go around telling people that forking the original is easy either - that would be very misleading. Much will have to be done to improve the current engine to make it easier to fork. I'm seriously considering doing that (or alternatively making a new benchmarks game from scratch with the intent of making it easy to configure and fork)
rdtsc 13y ago
That was there. PyPy was there at least. LuaJIT too I think. All go yanked out. The benchmarks are bogus anyway unless what you plan on working on n-body or fasta search or a short other list of things they run.
fabriceleal 13y ago
I believe Go is compiled [1][2]. Isn't a comparison with something like Haskell [3] more fair? [1] http://golang.org/doc/faq#Implementation http://golang.org/doc/faq#Implementation [2] http://golang.org/cmd/go/#hdr-Compile_and_run_Go_program http://golang.org/cmd/go/#hdr-Compile_and_run_Go_program [3] http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=go&lang2=ghc&data=u64 http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...
- elliotlai 13y ago
  The Glorious Glasgow Haskell Compilation System is truly glorious

sudhirj 13y ago

Why the spectacular loss to V8 in the Regex tests? Is Go's regex engine a naive implementation as of now?

chaosphere2112 13y ago
Russ Cox, in response to a regex benchmark vs. Python/Ruby. "You assume the benchmark is worth something. First of all, Ruby and Python are using C implementations of the regexp search, so Go is being beat by C, not by Ruby. Second, Go is using a different algorithm for regexp matching than the C implementations in those other languages. The algorithm Go uses guarantees to complete in time that is linear in the length of the input. The algorithm that Ruby/Python/etc are using can take time exponential in the length of the input, although on trivial cases it typically runs quite fast."[1] [1] https://groups.google.com/d/msg/golang-nuts/6d1maef4jQ8/yg4NY5qRtsMJ https://groups.google.com/d/msg/golang-nuts/6d1maef4jQ8/yg4N...
- baq 13y ago
  As a member of the masses of programmers that typically deal with truckloads of trivial problems, his reply is worth as much as the benchmark he criticizes. I've never used anything but a trivial regexp (for some values of trivial) in production code (one-shot problems don't count) and I don't give a damn what is being benchmarked here - I only care if my Python script runs fast.
  - ja30278 13y ago
    In this case, you really should care. http://www.benfrederickson.com/2013/04/09/python-catastrophic-regular-expressions-and-the-gil.html http://www.benfrederickson.com/2013/04/09/python-catastrophi...
  - kibwen 13y ago
    Actually, I might agree with Go here. I agree that most of my regexes are the trivial kind that don't exhibit exponential growth, but that's only because I also control the input to all those regexes. But Go's intended niche appears to be for user-facing server-side programs, and in that environment if you ever run a naive exponential-time regex against user-supplied input you're setting yourself up for a potential DOS attack.

cnlwsu 13y ago

Isn't comparing against a scripting language apples and oranges? Compiled vs interpreted one would assume to always look like this. Although the modern sophisticated JITs have definitely narrowed the playing field.

igouy 13y ago
http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.php#comparison http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...

zobzu 13y ago

I dunno about you but I don't use Go where I could use javascript. And also, I use multicore machines only. So I prefer this URL: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=go&lang2=gcc&data=u64q http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t... :P

tptacek 13y ago
Go vs. v8 is a reasonable comparison. Node.js is probably the closest substitute for the way Go is most often used today (as API servers or backend app components for web applications). I have no comment as to whether this "benchmarks game" stuff tells us anything meaningful.
burntsushi 13y ago
... That's using Go 1.0.3.
- mseepgood 13y ago
  Why are some benchmarks using Go 1.0.3 and some 1.1beta1? And why is the PCRE program broken? http://benchmarksgame.alioth.debian.org/u64/program.php?test=regexdna&lang=go&id=7 http://benchmarksgame.alioth.debian.org/u64/program.php?test... As far as I understand it just needs a "go get github.com/glenn-brown/golang-pkg-pcre/src/pkg/pcre"
  - igouy 13y ago
    https://news.ycombinator.com/item?id=5565769 https://news.ycombinator.com/item?id=5565769
    - mseepgood 13y ago
      Thanks.
xwei 13y ago
you prefer to do multi-thread programming in C? serious?

alecbenzer 13y ago

I'd be interested in seeing compile times. In a talk posted recently Rob Pike seemed to be kind of emphasizing #include inefficiencies and bytes read by compiler to bytes of source input ratios for C/C++, and how was better at this.