4 ms·

Also, looking at the white paper I see really weird numbers. This section in particular: Benchmarks and preliminary comparisons with Erlang, Scala, Akka, and l

Also, looking at the white paper I see really weird numbers. This section in particular: Benchmarks and preliminary comparisons with Erlang, Scala, Akka, and libccpa Table1 & Table2 It seems that you have the exact same numbers for Erlang and Scala in table2, this is very hard to believe. You either accidentally put down the same number twice or you measured the performance of your benchmarking tool, otherwise it is extremely unlikely that two entirely different systems give the the same exact measurement. Similar story in table1. maybe I am missing something terribly obvious but this looks off to me.

pedrocr 11y ago
>It seems that you have the exact same numbers for Erlang and Scala in table2, this is very hard to believe. The numbers are taken from another source and it seems they didn't have the exact numbers, hence the ~9s figure which is then used to calculate 333,333. They seem to have gotten the numbers by looking up the graphs on this page: http://libcppa.blogspot.co.uk/search/label/benchmark http://libcppa.blogspot.co.uk/search/label/benchmark Makes me wonder how precise you could be at getting the numbers from just the graphs. Pixel precision at 800x600 is probably not too bad.
- wpietri 11y ago
  I am not an expert in benchmarking, so maybe I'm missing something. But how is that not crazy? If I were looking to compare two things, I would run all the benchmarks on a single machine under my control. I might look at previously published reports to make sure I was getting comparable numbers. But there is no way I would publish a comparison that I merely hoped was apples to apples. I've just had too many benchmarks depend on subtle issues, ones that I had presumed were irrelevant.
  - decklebench 11y ago
    The benchmarks reported on the web page (http://ponylang.org http://ponylang.org) and also reported at http://ponylang.org/papers/fast-cheap.pdf http://ponylang.org/papers/fast-cheap.pdf are more recent, and for these the code was run for different languages, and on one machine.
- istvan__ 11y ago
  I am checking the source code of this benchmark and trying to reproduce the numbers.

pedrocr 11y ago

>It seems that you have the exact same numbers for Erlang and Scala in table2, this is very hard to believe. The numbers are taken from another source and it seems they didn't have the exact numbers, hence the ~9s figure which is then used to calculate 333,333. They seem to have gotten the numbers by looking up the graphs on this page: http://libcppa.blogspot.co.uk/search/label/benchmark http://libcppa.blogspot.co.uk/search/label/benchmark Makes me wonder how precise you could be at getting the numbers from just the graphs. Pixel precision at 800x600 is probably not too bad.

wpietri 11y ago
I am not an expert in benchmarking, so maybe I'm missing something. But how is that not crazy? If I were looking to compare two things, I would run all the benchmarks on a single machine under my control. I might look at previously published reports to make sure I was getting comparable numbers. But there is no way I would publish a comparison that I merely hoped was apples to apples. I've just had too many benchmarks depend on subtle issues, ones that I had presumed were irrelevant.
- decklebench 11y ago
  The benchmarks reported on the web page (http://ponylang.org http://ponylang.org) and also reported at http://ponylang.org/papers/fast-cheap.pdf http://ponylang.org/papers/fast-cheap.pdf are more recent, and for these the code was run for different languages, and on one machine.
istvan__ 11y ago
I am checking the source code of this benchmark and trying to reproduce the numbers.