5 ms·

Such programs are often specially and painstakingly constructed to avoid all the commonly used language features that are inefficient. For example, in Java, use

Such programs are often specially and painstakingly constructed to avoid all the commonly used language features that are inefficient. For example, in Java, user-defined data types are heap allocated and generic code boxes everything, even primitive types (an ArrayList of ints becomes unfortuately an array of pointers). Are these programs benchmarking typical idiomatic Java, or just some subset of the language?

CraigJPerry 5y ago
>> Are these programs benchmarking typical idiomatic Java https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/csharp.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... c# regex redux - 1.42 seconds java regex redux - 5.31 seconds Ok... but looking at the code: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-java-3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... import java.io.*; import java.util.*; import java.util.concurrent.CompletableFuture; import java.util.Map.Entry; import java.util.function.*; import java.util.regex.*; import static java.util.stream.Collectors.*; ... It's only using vanilla Java features. c# ? ... using System.Runtime.InteropServices; ... Interesting, why does it need that? [DllImport("pcre2-8", EntryPoint = "pcre2_compile_8", CharSet = CharSet.Ansi)] extern static IntPtr PcreCompile(string pattern, long length, uint options, out int errorcode, out long erroroffset, IntPtr ccontext); [DllImport("pcre2-8", EntryPoint = "pcre2_jit_compile_8", CharSet = CharSet.Ansi)] extern static int PcreJitCompile(IntPtr code, uint options); [DllImport("pcre2-8", EntryPoint = "pcre2_jit_match_8", CharSet = CharSet.Ansi)] extern unsafe static int PcreJitMatch(IntPtr code, byte* subject, long length, long startoffset, int options, IntPtr match_data, IntPtr mcontext); [DllImport("pcre2-8", EntryPoint = "pcre2_match_data_create_8", CharSet = CharSet.Ansi)] extern unsafe static IntPtr PcreMatchDataCreate(uint ovecsize, IntPtr mcontext); [DllImport("pcre2-8", EntryPoint = "pcre2_get_error_message_8", CharSet = CharSet.Ansi)] extern unsafe static int PcreGetErrorMessage(int errorcode, StringBuilder buffer, long bufflen); [DllImport("pcre2-8", EntryPoint = "pcre2_get_ovector_pointer_8", CharSet = CharSet.Ansi)] extern unsafe static IntPtr PcreGetOvectorPointer(IntPtr match_data); [DllImport("pcre2-8", EntryPoint = "pcre2_substitute_8", CharSet = CharSet.Ansi)] extern unsafe static int PcreSubstitute(IntPtr code, byte* subject, long length, long startoffset, int options, IntPtr match_data, IntPtr mcontext, byte* replacement, long rlength, byte* outputbuffer, out long outlength); Aha! It's because the c# impl is really just a wrapper round a native C impl of the problem. In what world is this a useful comparison? The fastest "real" c# solution is still faster than the java one though: c# (real) - 3.1 seconds https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-csharpcore-5.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- zigzag312 5y ago
  I agree. That's really not usefull comparision. They should create categories for each benchmark, like: - very naive code (shortest, most readable & easy to write code) - idiomatic code - optimized code without other-language-libs wrappers and without SIMD, single threaded - optimized code without other-language-libs wrappers and without SIMD, multi-threaded - optimized code without other-language-libs wrappers and with SIMD and/or multi-threaded - optimized code with other-language-libs wrappers allowed and any other optimization technique
  - igouy 5y ago
    You are free to take the data, create whatever categories you want and publish.
    - whthf_22 5y ago
      How would that undo the unfair comparison done in the benchmark game?
      - igouy 5y ago
        1. Not “unfair”. 2. We wouldn’t look at the benchmarks game if we thought an alternative presentation was more useful.
- grumpyprole 5y ago
  I agree it's not a useful comparison. That's why I don't give much weight to statements such as "Java comes on top in every individual category of all the benchmarks".
- igouy 5y ago
  > In what world is this a useful comparison? In this world where we also compare to C++ programs. In this world where — as you acknowledge — we also compare a C# regex-redux program that does not use a third party library. https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-csharpcore-5.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... > Interesting, why does it need that? 2 out of 10 tasks regex-redux and pidigits accept third-party libraries, the other 8 out of 10 tasks do not.
gameswithgo 5y ago
Yes any benchmark will be invalid for some people, such is life. If you want to claim or know something specific you will have to so your own painstaking investigation or find someone who has done that work. benchmarkgame does not attempt to compare idiomatic solutions for languages, it is closer to a “what is the best you can do” benchmark
- grumpyprole 5y ago
  > It is closer to a “what is the best you can do” benchmark As I suspected. So of course this tells us very little about how fast idiomatic code is relative to other languages. "The best I can do" is to invoke hand optimised assembly language, but rarely is that the right choice. A much more useful test would involve benchmarking some similar real world apps that solve the same problem.
  - igouy 5y ago
    Please show the objective rules that could be used to identify "idiomatic Java" and "idiomatic C#". Please show the objective rules to direct how comparison should be done when one languages "idiomatic" is not the same as some other languages "idiomatic" — to avoid you can write Java in any language.
vips7L 5y ago
Generic boxing should be fixed “soon” if they ever release Valhalla.
igouy 5y ago
Please show the objective rules that could be used to identify "typical idiomatic Java" and "typical idiomatic C#". Please show the objective rules to direct how comparison should be done when one languages "typical idiomatic" is not the same as some other languages "typical idiomatic" — to avoid you can write Java in any language.
- naasking 5y ago
  There's some low hanging fruit, like not permitting specialized collections in Java for a set of integers. Because of type erasure these are all heap allocated in Java but not in .NET. I think there's value in benchmarks showing both the fastest you can go if you need to (specializing everything to eke out max performance), and benchmarks showing how fast you will typically go if optimizing for productivity.
  - igouy 5y ago
    > … how fast you will typically go… Why would we think that would be similar for both you and `grumpyprole`.
    - naasking 5y ago
      You could probably constrain it sufficiently for some set of problems. Maybe something like: Solve problem X using the standard library associative map by elaborating the following pseudo code.
      - igouy 5y ago
        https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/knucleotide-java-4.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

CraigJPerry 5y ago

>> Are these programs benchmarking typical idiomatic Java https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/csharp.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... c# regex redux - 1.42 seconds java regex redux - 5.31 seconds Ok... but looking at the code: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-java-3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... import java.io.*; import java.util.*; import java.util.concurrent.CompletableFuture; import java.util.Map.Entry; import java.util.function.*; import java.util.regex.*; import static java.util.stream.Collectors.*; ... It's only using vanilla Java features. c# ? ... using System.Runtime.InteropServices; ... Interesting, why does it need that? [DllImport("pcre2-8", EntryPoint = "pcre2_compile_8", CharSet = CharSet.Ansi)] extern static IntPtr PcreCompile(string pattern, long length, uint options, out int errorcode, out long erroroffset, IntPtr ccontext); [DllImport("pcre2-8", EntryPoint = "pcre2_jit_compile_8", CharSet = CharSet.Ansi)] extern static int PcreJitCompile(IntPtr code, uint options); [DllImport("pcre2-8", EntryPoint = "pcre2_jit_match_8", CharSet = CharSet.Ansi)] extern unsafe static int PcreJitMatch(IntPtr code, byte* subject, long length, long startoffset, int options, IntPtr match_data, IntPtr mcontext); [DllImport("pcre2-8", EntryPoint = "pcre2_match_data_create_8", CharSet = CharSet.Ansi)] extern unsafe static IntPtr PcreMatchDataCreate(uint ovecsize, IntPtr mcontext); [DllImport("pcre2-8", EntryPoint = "pcre2_get_error_message_8", CharSet = CharSet.Ansi)] extern unsafe static int PcreGetErrorMessage(int errorcode, StringBuilder buffer, long bufflen); [DllImport("pcre2-8", EntryPoint = "pcre2_get_ovector_pointer_8", CharSet = CharSet.Ansi)] extern unsafe static IntPtr PcreGetOvectorPointer(IntPtr match_data); [DllImport("pcre2-8", EntryPoint = "pcre2_substitute_8", CharSet = CharSet.Ansi)] extern unsafe static int PcreSubstitute(IntPtr code, byte* subject, long length, long startoffset, int options, IntPtr match_data, IntPtr mcontext, byte* replacement, long rlength, byte* outputbuffer, out long outlength); Aha! It's because the c# impl is really just a wrapper round a native C impl of the problem. In what world is this a useful comparison? The fastest "real" c# solution is still faster than the java one though: c# (real) - 3.1 seconds https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-csharpcore-5.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

zigzag312 5y ago
I agree. That's really not usefull comparision. They should create categories for each benchmark, like: - very naive code (shortest, most readable & easy to write code) - idiomatic code - optimized code without other-language-libs wrappers and without SIMD, single threaded - optimized code without other-language-libs wrappers and without SIMD, multi-threaded - optimized code without other-language-libs wrappers and with SIMD and/or multi-threaded - optimized code with other-language-libs wrappers allowed and any other optimization technique
- igouy 5y ago
  You are free to take the data, create whatever categories you want and publish.
  - whthf_22 5y ago
    How would that undo the unfair comparison done in the benchmark game?
    - igouy 5y ago
      1. Not “unfair”. 2. We wouldn’t look at the benchmarks game if we thought an alternative presentation was more useful.
grumpyprole 5y ago
I agree it's not a useful comparison. That's why I don't give much weight to statements such as "Java comes on top in every individual category of all the benchmarks".
igouy 5y ago
> In what world is this a useful comparison? In this world where we also compare to C++ programs. In this world where — as you acknowledge — we also compare a C# regex-redux program that does not use a third party library. https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-csharpcore-5.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... > Interesting, why does it need that? 2 out of 10 tasks regex-redux and pidigits accept third-party libraries, the other 8 out of 10 tasks do not.

gameswithgo 5y ago

Yes any benchmark will be invalid for some people, such is life. If you want to claim or know something specific you will have to so your own painstaking investigation or find someone who has done that work. benchmarkgame does not attempt to compare idiomatic solutions for languages, it is closer to a “what is the best you can do” benchmark

grumpyprole 5y ago
> It is closer to a “what is the best you can do” benchmark As I suspected. So of course this tells us very little about how fast idiomatic code is relative to other languages. "The best I can do" is to invoke hand optimised assembly language, but rarely is that the right choice. A much more useful test would involve benchmarking some similar real world apps that solve the same problem.
- igouy 5y ago
  Please show the objective rules that could be used to identify "idiomatic Java" and "idiomatic C#". Please show the objective rules to direct how comparison should be done when one languages "idiomatic" is not the same as some other languages "idiomatic" — to avoid you can write Java in any language.

vips7L 5y ago

Generic boxing should be fixed “soon” if they ever release Valhalla.

igouy 5y ago

Please show the objective rules that could be used to identify "typical idiomatic Java" and "typical idiomatic C#". Please show the objective rules to direct how comparison should be done when one languages "typical idiomatic" is not the same as some other languages "typical idiomatic" — to avoid you can write Java in any language.

naasking 5y ago
There's some low hanging fruit, like not permitting specialized collections in Java for a set of integers. Because of type erasure these are all heap allocated in Java but not in .NET. I think there's value in benchmarks showing both the fastest you can go if you need to (specializing everything to eke out max performance), and benchmarks showing how fast you will typically go if optimizing for productivity.
- igouy 5y ago
  > … how fast you will typically go… Why would we think that would be similar for both you and `grumpyprole`.
  - naasking 5y ago
    You could probably constrain it sufficiently for some set of problems. Maybe something like: Solve problem X using the standard library associative map by elaborating the following pseudo code.
    - igouy 5y ago
      https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/knucleotide-java-4.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...