3 ms·

A few days ago I was checking the former computer language shootout for another language and saw that C# is beating Java on all the benchmarks. This surprised m

A few days ago I was checking the former computer language shootout for another language and saw that C# is beating Java on all the benchmarks. This surprised me because I always had remembered Java being slightly ahead. https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/csharp.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Can someone more knowledgeable give me some perspective on how representative these benchmarks are? At this point would it be wrong to say that in general the C# and .net environment is more performant than Java? If so, will there be any changes to Java to bridge any gap?

nicoburns 8y ago
I've seen multiple reports of large performance improvements (like ~30%) when moving from the old .Net to .Net Core (the newer, open source version). I imagine this would be more than enough of a difference to overtake Java. C# has a number of features (like `Span<T>` and value types) which Java doesn't currently have, which enable programs to run more efficiently, which might explain the difference. IIRC there are plans to add value types to Java, but they're not ready yet.
codexon 8y ago
I think the .net implementation benchmarked before was using mono, that's why Java was ahead before.
rlp 8y ago
Well, I would say that it highly depends on your workload. .NET's biggest performance win is lower-level memory management, like structs and Spans. If you can use those to avoid allocations and improve memory locality in your critical path, you'll end up beating the JVM. For allocation-heavy workloads that can't use .NET's lower-level memory tools, I think the JVM is still a bit ahead with its better GC and profiling HotSpot JIT compiler.
nqzero 8y ago
that benchmark is terrible (see the github for details). for k-nucleotides, the test that showed the biggest delta (c# vs java), i was able to improve the java perf by 35%. but the test harness is so poorly done that i didn't bother running it against the other impls. the reality is that good java developers just don't care about this sort of thing (which may say something unfortunate about me) and the benchmark is bad https://github.com/nqzero/k-nucleotide https://github.com/nqzero/k-nucleotide
- igouy 8y ago
  > able to improve the java perf by 35% How much more memory did you use? > i didn't bother running it https://salsa.debian.org/benchmarksgame-team/benchmarksgame/blob/master/CONTRIBUTING.md https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
- igouy 8y ago
  What a terrible attempt at criticism — `nqzero` says it's terrible so it must be terrible! `nqzero` says "the test harness is difficult to configure and use" — but does not show any example of what is supposed to be difficult about using the bencher script. `nqzero` says "the tests are not representative of common programming tasks" — but does not show any example common programming tasks and does not show the tests are not representative. `nqzero` says "there's no attempt to account for JIT warmup, and many of the tasks are too short to ever warm up" — but admits that "it is plausible" warmup costs are amortised and comparison will show miniscule difference. `nqzero` says "maintainers are opinionated in terms of what code they'll allow, effectively choosing the winners" — but (again) does not show any example. `nqzero` says "doesn't appear to allow for jvm options to be included" — but seems not to have looked. `nqzero` says "the test cpu is from 2007 and is not necessarily representative of current cpus" — That at-least is true!
benaadams 8y ago
More coming in .NET Core 3.0 as it get access to all the SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX, AVX2 and FMA cpu intrinsics https://github.com/dotnet/corefx/tree/master/src/Common/src/CoreLib/System/Runtime/Intrinsics https://github.com/dotnet/corefx/tree/master/src/Common/src/...
int_19h 8y ago
Historically, Java has better JIT compilers. One of the reasons is that Java started as a bytecode interpreter, and JIT was later bolted on for hot paths - this makes it possible to have JIT run for longer (since you can interpret bytecode in the meantime). .NET was JIT-only from the get go though, and its JIT compiler has to be "fast enough" that cold calls aren't atrociously slow, which restricts optimizations. On the other hand, C# has features like structs, non-virtual methods (by default at that), and reified generics, that make for faster code without any JIT magic. And, depending on the exact code being tested, one or the other factor can easily dominate.
runfaster2000 8y ago
.NET team member here. We haven't invested any effort into this particular benchmark. We're happy to invest in benchmarks that have well-defined (and enforced) rules on entries. Our understanding is that the "game" part of "benchmarkgame" is appropriately descriptive. We have put a lot of effort into the TechEmpower benchmarks. In terms of .NET performance, we've made significant improvements [1] with each release. If there was a gap between C# and other languages before, it is likely that it is much smaller now due to our efforts. We have a balanced view on performance. We are investing in application throughput, latency, memory usage and disk footprint. Most .NET users care about multiple of those things. We also hear from users who care disproportionately about just one of them. This motivates us to care about all of these things, although throughput and memory usage is where the biggest gains are. I have this (not very good) joke that many .NET developers until recently would not have been able to accurately describe what structs were (as opposed to classes). To a large degree this was because we used them in very narrow ways in the products. That has changed a lot in the last few years. ValueTuple, ValueTask and Span are all good examples. The culture of the .NET has changed and so is the .NET community around us. This will result in higher performance code more generally within typical .NET applications (as opposed to some theoretical outcome). If this happens, and I do believe it will, .NET will be forever changed in a pretty radical (or at least "rad") way. [1] https://blogs.msdn.microsoft.com/dotnet/2018/04/18/performance-improvements-in-net-core-2-1/ https://blogs.msdn.microsoft.com/dotnet/2018/04/18/performan...

nicoburns 8y ago

I've seen multiple reports of large performance improvements (like ~30%) when moving from the old .Net to .Net Core (the newer, open source version). I imagine this would be more than enough of a difference to overtake Java. C# has a number of features (like `Span<T>` and value types) which Java doesn't currently have, which enable programs to run more efficiently, which might explain the difference. IIRC there are plans to add value types to Java, but they're not ready yet.

codexon 8y ago

I think the .net implementation benchmarked before was using mono, that's why Java was ahead before.

rlp 8y ago

Well, I would say that it highly depends on your workload. .NET's biggest performance win is lower-level memory management, like structs and Spans. If you can use those to avoid allocations and improve memory locality in your critical path, you'll end up beating the JVM. For allocation-heavy workloads that can't use .NET's lower-level memory tools, I think the JVM is still a bit ahead with its better GC and profiling HotSpot JIT compiler.

nqzero 8y ago

that benchmark is terrible (see the github for details). for k-nucleotides, the test that showed the biggest delta (c# vs java), i was able to improve the java perf by 35%. but the test harness is so poorly done that i didn't bother running it against the other impls. the reality is that good java developers just don't care about this sort of thing (which may say something unfortunate about me) and the benchmark is bad https://github.com/nqzero/k-nucleotide https://github.com/nqzero/k-nucleotide

igouy 8y ago
> able to improve the java perf by 35% How much more memory did you use? > i didn't bother running it https://salsa.debian.org/benchmarksgame-team/benchmarksgame/blob/master/CONTRIBUTING.md https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
igouy 8y ago
What a terrible attempt at criticism — `nqzero` says it's terrible so it must be terrible! `nqzero` says "the test harness is difficult to configure and use" — but does not show any example of what is supposed to be difficult about using the bencher script. `nqzero` says "the tests are not representative of common programming tasks" — but does not show any example common programming tasks and does not show the tests are not representative. `nqzero` says "there's no attempt to account for JIT warmup, and many of the tasks are too short to ever warm up" — but admits that "it is plausible" warmup costs are amortised and comparison will show miniscule difference. `nqzero` says "maintainers are opinionated in terms of what code they'll allow, effectively choosing the winners" — but (again) does not show any example. `nqzero` says "doesn't appear to allow for jvm options to be included" — but seems not to have looked. `nqzero` says "the test cpu is from 2007 and is not necessarily representative of current cpus" — That at-least is true!

benaadams 8y ago

More coming in .NET Core 3.0 as it get access to all the SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX, AVX2 and FMA cpu intrinsics https://github.com/dotnet/corefx/tree/master/src/Common/src/CoreLib/System/Runtime/Intrinsics https://github.com/dotnet/corefx/tree/master/src/Common/src/...

int_19h 8y ago

Historically, Java has better JIT compilers. One of the reasons is that Java started as a bytecode interpreter, and JIT was later bolted on for hot paths - this makes it possible to have JIT run for longer (since you can interpret bytecode in the meantime). .NET was JIT-only from the get go though, and its JIT compiler has to be "fast enough" that cold calls aren't atrociously slow, which restricts optimizations. On the other hand, C# has features like structs, non-virtual methods (by default at that), and reified generics, that make for faster code without any JIT magic. And, depending on the exact code being tested, one or the other factor can easily dominate.

runfaster2000 8y ago

.NET team member here. We haven't invested any effort into this particular benchmark. We're happy to invest in benchmarks that have well-defined (and enforced) rules on entries. Our understanding is that the "game" part of "benchmarkgame" is appropriately descriptive. We have put a lot of effort into the TechEmpower benchmarks. In terms of .NET performance, we've made significant improvements [1] with each release. If there was a gap between C# and other languages before, it is likely that it is much smaller now due to our efforts. We have a balanced view on performance. We are investing in application throughput, latency, memory usage and disk footprint. Most .NET users care about multiple of those things. We also hear from users who care disproportionately about just one of them. This motivates us to care about all of these things, although throughput and memory usage is where the biggest gains are. I have this (not very good) joke that many .NET developers until recently would not have been able to accurately describe what structs were (as opposed to classes). To a large degree this was because we used them in very narrow ways in the products. That has changed a lot in the last few years. ValueTuple, ValueTask and Span are all good examples. The culture of the .NET has changed and so is the .NET community around us. This will result in higher performance code more generally within typical .NET applications (as opposed to some theoretical outcome). If this happens, and I do believe it will, .NET will be forever changed in a pretty radical (or at least "rad") way. [1] https://blogs.msdn.microsoft.com/dotnet/2018/04/18/performance-improvements-in-net-core-2-1/ https://blogs.msdn.microsoft.com/dotnet/2018/04/18/performan...