7 ms·

C# .NET Core versus Java fastest programs

100 points by throwaway13337 6y ago

5986043handy 6y ago
Take with a grain of salt, JVM tuning is an entire art and science, probably same with CLR.
- adamnemecek 6y ago
  Presumably, these are comparably tuned.
  - bremac 6y ago
    The .net benchmarks use architecture-specific SIMD intrinsics. There's no equivalent on the JVM, though JEP 338 aims to add a vector API.
deleted 6y ago
[deleted]
xiphias2 6y ago
Just looking at some .NET programs, they all use unsafe code, which is always faster than using garbage collected memory management, so the results aren't surprising.
- cratermoon 6y ago
  Gaming the benchmarks for fun and profit.
- deleted 6y ago
  [deleted]
- zelly 6y ago
  It's using features of the language. Is that not what this is meant to measure/compare? .NET has unsafe, SIMD, and value types which opens it to a whole world of programming that Java is currently unsuited for.
  - derefr 6y ago
    One would think that a benchmark described in reference to the performance of the virtual machines themselves, would be measuring the performance of pure bytecode (or a JITing thereof), not the performance of native code linked into the runtime.
    - jashmatthews 6y ago
      Using a native regex extension lib is the norm for "scripting" languages so totally reasonable to benchmark like that. How do you define when it's OK to use a native extension and when it's cheating?
  - cel1ne 6y ago
    You can call C with Java.
    - igouy 6y ago
      For example https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/pidigits-java-3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
  - fomine3 6y ago
    Benchmarking both C# [Managed] and C# [Unsafe] is preferred.
    - igouy 6y ago
      Like this ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/csharpcore.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- john_moscow 6y ago
  If you want performance, you need to refactor your code into a small performance-critical unsafe core (that has higher test coverage and audit standards), while keeping the rest of the business logic in the safe domain, and relying on the CLR to do the heavy lifting and guarantee memory consistency. Bonus point: you can write the core in C++ and use COM-like marshaling for absolutely unbeatable performance (no actual COM runtime is needed).
- igouy 6y ago
  Maybe look at the .NET programs that do not use unsafe code? https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/csharpcore.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
exabrial 6y ago
These are pretty hilarious.
- exabrial 6y ago
  Did you guys actually read the code in the benchmarks before downvoting me? Both sides are terrible.
  - igouy 6y ago
    Please be specific.
shrewduser 6y ago
I took a quick look at the code, it's nothing alike. It seems they have people submit solutions to the problem and compare the fastest implementation for each.
- bArray 6y ago
  And the fastest implementations aren't the fastest either, I was able to knock about 10% off one of the Java examples from 2010. I suspect the C# folks were a lot more motivated to "prove" that C# was no longer slow. As others have pointed out, if you want to test C#'s ability to call C-functions in an unsafe way, then fair enough. Otherwise, with memory safety enabled, they are roughly comparable. That said, I did port the C# version to Java and it added about 20% extra time, even with some of the cheats (algorithm optimizations) it has implemented.
  - igouy 6y ago
    > I suspect the C# folks were a lot more motivated to "prove" that C# was no longer slow. One person became curious and was brave enough to contribute their programs.
zucker42 6y ago
I thought I remember a post about them shutting it down from a few years back. Did they reopen it, or did I just fabricate that memory?
- philliphaydon 6y ago
  Shutting what down?
  - noble_pleb 6y ago
    The debian benchmarks website. It was called benchmarkgame.alioth.com earlier if I recall correctly.
    - philliphaydon 6y ago
      I didn’t even realise the domain was different. Been so many years since I’ve seen this site I had no idea it was gone.
      - mappu 6y ago
        Alioth was the name of a machine running FusionForge. Debian replaced it with "Salsa" that's running GitLab. The benchmarks game is hosted in this repository: https://salsa.debian.org/benchmarksgame-team/benchmarksgame https://salsa.debian.org/benchmarksgame-team/benchmarksgame But (without intending to spread unsubstantiated rumor) it seems like it's temporarily unmaintained at the moment: - in the README.md there is some complaint about being difficult to reboot into a Linux dual-boot, maybe the measurements+website were being published manually - CI yml seems incomplete - Some very strange issues like https://salsa.debian.org/benchmarksgame-team/benchmarksgame/-/issues/347 https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
        zelly 6y ago
        This is really ghetto. Why does Debian allow it on their domain?
magicalhippo 6y ago
Only a few seems to be plain code vs plain code. For the C# code there's use of unsafe, SSE/AVX intrinsics and calling C code to do the job (PCRE for the regex). Certainly it's nice that you have these capabilities in C#, but at least for me it limits the usefulness of the results.
- zamalek 6y ago
  Handicapping one language to the featureset of another is unreasonable. You'd also have to disable a host of JVM features (such as hotspot) for a "fair" comparison, which is ridiculous. Calling C is unfair, though. JVM can do the same and it should be implemented the same way.
  - bitbckt 6y ago
    Referring to Hotspot as a “feature” of the JVM is a gross mischaracterization. A JVM is not Hotspot, and Hotspot is not a part of “the” JVM.
    - nix23 6y ago
      BS! The JVM from Oracle/Sun is called "Java HotSpot Performance Engine" in short HotSpot: https://en.wikipedia.org/wiki/HotSpot https://en.wikipedia.org/wiki/HotSpot
      - bitbckt 6y ago
        One JVM is, yes. That’s exactly my point.
  - jsiepkes 6y ago
    > Calling C is unfair, though. JVM can do the same and it should be implemented the same way. If you want to benchmark C code you can just benchmark a C application. You run stuff in a VM like the JVM or CLR because you want things like memory safety. If you allow native code in something like a regex benchmark then all your benchmarks are basically about: How fast can the JVM/CLR call native code. And say absolutely nothing about how fast the VM itself is. So using native code in these benchmarks is just silly and basically makes the whole thing meaningless.
runfaster2000 6y ago
.NET Team member here. We don't look at these specific benchmarks. Our belief is that they are wild west w/limited moderation of what is reasonable/real-world and not. Not necessarily a reasonable comparison, but we focus on TechEmpower[1]. You can check out the latest .NET performance improvements in our latest blog post on the topic [2]. [1] https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ [2] https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-5/ https://devblogs.microsoft.com/dotnet/performance-improvemen...
- jonathanoliver 6y ago
  I am thoroughly impressed with you what your team has accomplished. Roughly 7-8 years ago, the company I work with switched away from .NET (to Go) largely because our performance with IIS for HTTP traffic running .NET was...poor by comparison to other available options. I wrote a blog post[1] on the subject. It's been exciting to watch the hard work you and your team are investing to make .NET (Core/Framework/etc) faster. In fact, virtually every single issue or point I raised in my blog post has been addressed to my sanctification and then some. The Techempower benchmarks show the massive strides you've all made and now I'm delighted to see .NET consistently performing at or near the top of the benchmark even without heavy customization/modification. To this day, I still really love C# as a language and there are a number of significant things in C# that I wish the Go ecosystem would adopt. If/when bootstrap a technology stack again, you can bet that C#/.NET will be the top contender on the list of options. Thanks again for all of your effort. It is very much appreciated and admired, even if I'm not using that tech stack at this point. [1] https://blog.jonathanoliver.com/why-i-left-dot-net/ https://blog.jonathanoliver.com/why-i-left-dot-net/
  - benaadams 6y ago
    Are Go vs C# results there too https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/go-csharpcore.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Go being only faster in one benchmark
  - whycombagator 6y ago
smabie 6y ago
Everyone just games these benchmarks by calling out to C libraries. It's better to actually compare the slowest programs for each language instead of the fastest ones, because those are usually idiomatic.
- igouy 6y ago
  > Everyone just games these benchmarks by calling out to C libraries. Really? Please say which of the n-body programs do that. https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Meanwhile, sort by program source code size — https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody-gz.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
est31 6y ago
Seeing the debian.net domain name, I'm a bit saddened to be reminded that DotNet Core is open source, yet still not available on Debian without installing packages from external sources (like the deb packages MS provides): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779970 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779970 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779969 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779969 Right now the official install instructions want you to trust Microsoft keys and I'm not really a fan of that. Further, it installs a custom package repo and in the past I've had very negative experiences with PPAs so I tend to avoid them. https://docs.microsoft.com/en-us/dotnet/core/install/linux-debian#debian-10- https://docs.microsoft.com/en-us/dotnet/core/install/linux-d...
- moron4hire 6y ago
  > Right now the official install instructions want you to trust Microsoft keys and I'm not really a fan of that. Why?
  - zelly 6y ago
    Because people in the SV/HN bubble hate Microsoft. Purely for cultural reasons of course (while running Facebook's Pytorch, Google's browser, Apple's laptop, etc.).
    - mbreese 6y ago
      I think you’ll find plenty of complaints with Facebook, Google, and Apple mixed into their relevant conversations. In fact, I’m pretty sure many people on here would have issues installing a key from Facebook too. Admittedly though, it’s hard to avoid a Google or Apple key, if you’re using an OS from them...
      - est31 6y ago
        Personally, I'm sort of a purist so while I use Google's Android OS, I use the Lineage OS distro without gapps, which means I'm one of the few Android users without Google's keys (at least for the gapps). And yeah, I'd have issues with installing pytorch too. I tried it once, but then gave up because I couldn't ensure it wouldn't mess up my system and I mainly wanted to try out some ML stuff on Github.
axaxs 6y ago
JVM clone edges out JVM. Fantastic.
chromedev 6y ago
So do Go, C, and many others.
sercand 6y ago
Unity also shows how you can use C# for high performance game code with their custom burst compiler. They even switch some of their C++ code to C#.
- nix23 6y ago
  You can feel the sluggishness of every Unity game instantly, not saing it's C# fault, but its not a good engine for fast games.
- Churb 6y ago
  They’re moving away from the managed runtimes and compiling straight to native code — it’s not really comparable to what is being discussed here.
tyingq 6y ago
The regex-redux one looks like more of an endorsement for the PCRE2 jit.
- igouy 6y ago
  Like compare `regex-redux #8` and `regex-redux #5` ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/csharpcore.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Churb 6y ago
.NET core was slower than mono which was slower than JVM for running real time multiplayer games (3d math, serialization, reading and writing from multiple streams) as of a couple of years ago, did something big change With the runtimes very recently or is this just coding to benchmarks?
Dork1234 6y ago
Maybe I missed something, but there doesn't appear to be a benchmark between JNI, JNA, and pInvoke. Most my experience with .NET/Java has shown it is fast enough in most cases, but then run into areas that need to be optimized into Native Code. This calling of Native "C" code can be slow with JNI and generally just fast enough for .NET pInvoke. I haven't ever had a reason to test JNA.

5986043handy 6y ago

Take with a grain of salt, JVM tuning is an entire art and science, probably same with CLR.

adamnemecek 6y ago
Presumably, these are comparably tuned.
- bremac 6y ago
  The .net benchmarks use architecture-specific SIMD intrinsics. There's no equivalent on the JVM, though JEP 338 aims to add a vector API.

deleted 6y ago

[deleted]

xiphias2 6y ago

Just looking at some .NET programs, they all use unsafe code, which is always faster than using garbage collected memory management, so the results aren't surprising.

cratermoon 6y ago
Gaming the benchmarks for fun and profit.
deleted 6y ago
[deleted]
zelly 6y ago
It's using features of the language. Is that not what this is meant to measure/compare? .NET has unsafe, SIMD, and value types which opens it to a whole world of programming that Java is currently unsuited for.
- derefr 6y ago
  One would think that a benchmark described in reference to the performance of the virtual machines themselves, would be measuring the performance of pure bytecode (or a JITing thereof), not the performance of native code linked into the runtime.
  - jashmatthews 6y ago
    Using a native regex extension lib is the norm for "scripting" languages so totally reasonable to benchmark like that. How do you define when it's OK to use a native extension and when it's cheating?
- cel1ne 6y ago
  You can call C with Java.
  - igouy 6y ago
    For example https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/pidigits-java-3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- fomine3 6y ago
  Benchmarking both C# [Managed] and C# [Unsafe] is preferred.
  - igouy 6y ago
    Like this ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/csharpcore.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
john_moscow 6y ago
If you want performance, you need to refactor your code into a small performance-critical unsafe core (that has higher test coverage and audit standards), while keeping the rest of the business logic in the safe domain, and relying on the CLR to do the heavy lifting and guarantee memory consistency. Bonus point: you can write the core in C++ and use COM-like marshaling for absolutely unbeatable performance (no actual COM runtime is needed).
igouy 6y ago
Maybe look at the .NET programs that do not use unsafe code? https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/csharpcore.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

exabrial 6y ago

These are pretty hilarious.

exabrial 6y ago
Did you guys actually read the code in the benchmarks before downvoting me? Both sides are terrible.
- igouy 6y ago
  Please be specific.

shrewduser 6y ago

I took a quick look at the code, it's nothing alike. It seems they have people submit solutions to the problem and compare the fastest implementation for each.

bArray 6y ago
And the fastest implementations aren't the fastest either, I was able to knock about 10% off one of the Java examples from 2010. I suspect the C# folks were a lot more motivated to "prove" that C# was no longer slow. As others have pointed out, if you want to test C#'s ability to call C-functions in an unsafe way, then fair enough. Otherwise, with memory safety enabled, they are roughly comparable. That said, I did port the C# version to Java and it added about 20% extra time, even with some of the cheats (algorithm optimizations) it has implemented.
- igouy 6y ago
  > I suspect the C# folks were a lot more motivated to "prove" that C# was no longer slow. One person became curious and was brave enough to contribute their programs.

zucker42 6y ago

I thought I remember a post about them shutting it down from a few years back. Did they reopen it, or did I just fabricate that memory?

philliphaydon 6y ago
Shutting what down?
- noble_pleb 6y ago
  The debian benchmarks website. It was called benchmarkgame.alioth.com earlier if I recall correctly.
  - philliphaydon 6y ago
    I didn’t even realise the domain was different. Been so many years since I’ve seen this site I had no idea it was gone.
    - mappu 6y ago
      Alioth was the name of a machine running FusionForge. Debian replaced it with "Salsa" that's running GitLab. The benchmarks game is hosted in this repository: https://salsa.debian.org/benchmarksgame-team/benchmarksgame https://salsa.debian.org/benchmarksgame-team/benchmarksgame But (without intending to spread unsubstantiated rumor) it seems like it's temporarily unmaintained at the moment: - in the README.md there is some complaint about being difficult to reboot into a Linux dual-boot, maybe the measurements+website were being published manually - CI yml seems incomplete - Some very strange issues like https://salsa.debian.org/benchmarksgame-team/benchmarksgame/-/issues/347 https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
      - zelly 6y ago
        This is really ghetto. Why does Debian allow it on their domain?

magicalhippo 6y ago

Only a few seems to be plain code vs plain code. For the C# code there's use of unsafe, SSE/AVX intrinsics and calling C code to do the job (PCRE for the regex). Certainly it's nice that you have these capabilities in C#, but at least for me it limits the usefulness of the results.

zamalek 6y ago
Handicapping one language to the featureset of another is unreasonable. You'd also have to disable a host of JVM features (such as hotspot) for a "fair" comparison, which is ridiculous. Calling C is unfair, though. JVM can do the same and it should be implemented the same way.
- bitbckt 6y ago
  Referring to Hotspot as a “feature” of the JVM is a gross mischaracterization. A JVM is not Hotspot, and Hotspot is not a part of “the” JVM.
  - nix23 6y ago
    BS! The JVM from Oracle/Sun is called "Java HotSpot Performance Engine" in short HotSpot: https://en.wikipedia.org/wiki/HotSpot https://en.wikipedia.org/wiki/HotSpot
    - bitbckt 6y ago
      One JVM is, yes. That’s exactly my point.
- jsiepkes 6y ago
  > Calling C is unfair, though. JVM can do the same and it should be implemented the same way. If you want to benchmark C code you can just benchmark a C application. You run stuff in a VM like the JVM or CLR because you want things like memory safety. If you allow native code in something like a regex benchmark then all your benchmarks are basically about: How fast can the JVM/CLR call native code. And say absolutely nothing about how fast the VM itself is. So using native code in these benchmarks is just silly and basically makes the whole thing meaningless.

runfaster2000 6y ago

.NET Team member here. We don't look at these specific benchmarks. Our belief is that they are wild west w/limited moderation of what is reasonable/real-world and not. Not necessarily a reasonable comparison, but we focus on TechEmpower[1]. You can check out the latest .NET performance improvements in our latest blog post on the topic [2]. [1] https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ [2] https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-5/ https://devblogs.microsoft.com/dotnet/performance-improvemen...

jonathanoliver 6y ago
I am thoroughly impressed with you what your team has accomplished. Roughly 7-8 years ago, the company I work with switched away from .NET (to Go) largely because our performance with IIS for HTTP traffic running .NET was...poor by comparison to other available options. I wrote a blog post[1] on the subject. It's been exciting to watch the hard work you and your team are investing to make .NET (Core/Framework/etc) faster. In fact, virtually every single issue or point I raised in my blog post has been addressed to my sanctification and then some. The Techempower benchmarks show the massive strides you've all made and now I'm delighted to see .NET consistently performing at or near the top of the benchmark even without heavy customization/modification. To this day, I still really love C# as a language and there are a number of significant things in C# that I wish the Go ecosystem would adopt. If/when bootstrap a technology stack again, you can bet that C#/.NET will be the top contender on the list of options. Thanks again for all of your effort. It is very much appreciated and admired, even if I'm not using that tech stack at this point. [1] https://blog.jonathanoliver.com/why-i-left-dot-net/ https://blog.jonathanoliver.com/why-i-left-dot-net/
- benaadams 6y ago
  Are Go vs C# results there too https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/go-csharpcore.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Go being only faster in one benchmark
- whycombagator 6y ago

smabie 6y ago

Everyone just games these benchmarks by calling out to C libraries. It's better to actually compare the slowest programs for each language instead of the fastest ones, because those are usually idiomatic.

igouy 6y ago
> Everyone just games these benchmarks by calling out to C libraries. Really? Please say which of the n-body programs do that. https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Meanwhile, sort by program source code size — https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody-gz.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

est31 6y ago

Seeing the debian.net domain name, I'm a bit saddened to be reminded that DotNet Core is open source, yet still not available on Debian without installing packages from external sources (like the deb packages MS provides): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779970 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779970 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779969 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779969 Right now the official install instructions want you to trust Microsoft keys and I'm not really a fan of that. Further, it installs a custom package repo and in the past I've had very negative experiences with PPAs so I tend to avoid them. https://docs.microsoft.com/en-us/dotnet/core/install/linux-debian#debian-10- https://docs.microsoft.com/en-us/dotnet/core/install/linux-d...

moron4hire 6y ago
> Right now the official install instructions want you to trust Microsoft keys and I'm not really a fan of that. Why?
- zelly 6y ago
  Because people in the SV/HN bubble hate Microsoft. Purely for cultural reasons of course (while running Facebook's Pytorch, Google's browser, Apple's laptop, etc.).
  - mbreese 6y ago
    I think you’ll find plenty of complaints with Facebook, Google, and Apple mixed into their relevant conversations. In fact, I’m pretty sure many people on here would have issues installing a key from Facebook too. Admittedly though, it’s hard to avoid a Google or Apple key, if you’re using an OS from them...
    - est31 6y ago
      Personally, I'm sort of a purist so while I use Google's Android OS, I use the Lineage OS distro without gapps, which means I'm one of the few Android users without Google's keys (at least for the gapps). And yeah, I'd have issues with installing pytorch too. I tried it once, but then gave up because I couldn't ensure it wouldn't mess up my system and I mainly wanted to try out some ML stuff on Github.

axaxs 6y ago

JVM clone edges out JVM. Fantastic.

chromedev 6y ago

So do Go, C, and many others.

sercand 6y ago

Unity also shows how you can use C# for high performance game code with their custom burst compiler. They even switch some of their C++ code to C#.

nix23 6y ago
You can feel the sluggishness of every Unity game instantly, not saing it's C# fault, but its not a good engine for fast games.
Churb 6y ago
They’re moving away from the managed runtimes and compiling straight to native code — it’s not really comparable to what is being discussed here.

tyingq 6y ago

The regex-redux one looks like more of an endorsement for the PCRE2 jit.

igouy 6y ago
Like compare `regex-redux #8` and `regex-redux #5` ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/csharpcore.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Churb 6y ago

.NET core was slower than mono which was slower than JVM for running real time multiplayer games (3d math, serialization, reading and writing from multiple streams) as of a couple of years ago, did something big change With the runtimes very recently or is this just coding to benchmarks?

Dork1234 6y ago

Maybe I missed something, but there doesn't appear to be a benchmark between JNI, JNA, and pInvoke. Most my experience with .NET/Java has shown it is fast enough in most cases, but then run into areas that need to be optimized into Native Code. This calling of Native "C" code can be slow with JNI and generally just fast enough for .NET pInvoke. I haven't ever had a reason to test JNA.