5 ms·

Those are notoriously cheat-y for just about all languages on that list. Any actual project is never going to get close to the efficiency that those number woul

by EraYaN 2y ago

Those are notoriously cheat-y for just about all languages on that list. Any actual project is never going to get close to the efficiency that those number would require. Both Java and .NET over promise by A LOT with the numbers they get there.

Ygg2 2y ago
Cheat-y in what way? I don't consider micro benchmarks that interesting especially since C# does have two aces that Java will get in mid term - namely Simd and primitive types. The Tech Empower benchmarks do seem to reflect general state of Java Web Framework ecosystem with Vert.x being they hyper fast web framework and Spring being way slower.
- EraYaN 2y ago
  Well the amount of tricks that basically nobody can implement in production that those implementations do to get the numbers that they do is ridiculous. Meaning the numbers they get are wildly optimistic at best and misleading at worst. If you take the standard template for any of these frameworks (both Java and C# and any other language) and you add authentication etc, the real performance will be 5-10% of the numbers reported in those benchmarks. Have a look through some of the weirdness in the implementations it's wild (and sometimes educational). The .NET team especially has done stuff specifically to get faster on those benchmarks.
  - neonsunset 2y ago
    I second this sentiment. If anything, I genuinely think that the way .NET's TechEmpower submissions look does more damage than good. BenchmarksGame offers a much better close-up comparison by having much simpler submissions that mostly consist of code that you could totally see yourself write in a regular setting. .NET is perfectly capable of standing on its own, and if there are specific areas that need improvement - this should serve as a push to further improve DB driver implementations and make ASP.NET Core more robust against various feature configurations. It is already much, much faster than Spring which is a good start, but it could be pushed further. I'd like to note that neither Go nor Java are viable for high-performance programming in a way that C# is. Neither gives you the required low-level access, performance oriented APIs, ability to use zero-cost abstractions and platform control you get to have in .NET. You can get far with both, but not C++/Rust-far the way you can with C#.
    - Ygg2 2y ago
      > BenchmarksGame offers a much better close-up comparison by having much simpler submissions that mostly consist of code that you could totally see yourself write in a regular setting Yeah, except if you are working on Web servers the quality of the framework and its supporting libraries is much more important than what code could theoretically achieve. What is the point of being able to 200 mph when you only ever drive up to 30mph. > Neither gives you the required low-level access, performance oriented APIs, ability to use zero-cost abstractions. Java is working on high performance abstractions, see Vector API (Simd) and project Valhalla (custom primitive types). Sure C# has a theoretical leg up (for which it paid dearly by causing backwards incompatibility with reified generics) but most of the libraries don't use low-level access or SIMD optimizations or what not.
  - dahauns 2y ago
    > Have a look through some of the weirdness in the implementations it's wild (and sometimes educational). The .NET team especially has done stuff specifically to get faster on those benchmarks. Could you give me a pointer or two? I wondered about that myself, especially considering the massive improvement from "old" .NET to the core/kestrel based solutions - but a quick browsing a while ago mostly left me astonished how...well, for lack of a better word, banal most of the code was. Agreed though, lack of all kinds of layers like auth, orm etc. are sadly a drawback of these kinds of benchmarks, if understandable - it would make comparability even trickier and has the danger of the comparison matrix of systems/frameworks/libraries exploding in size. But yeah, would be nice datapoints to have. :)
    - EraYaN 2y ago
      They don't even use Razor Pages but a custom RazorSlices package to do the templating [1]. Yes, that is much faster because it removes MVC and a ton of infrastructure but it's also kind of gross. Also the use of stuff like UnsafePreferInlineScheduling has some downsides (running application code on the IO thread) and honestly I'd never use in production. The custom BufferWriter stuff is pretty neat though, although also not really something most people will reach for. And there is more, like the caching of StringBuilders etc. But it also doesn't use the actual HTTP server to build headers, but they just dump a string into the socket [2], feels a bit unrealistic to me. In general the BenchmarkApplication class [3] is full of non-standard stuff that you'd normally let the framework handle. [1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/src/Platform/Templates/FortunesUtf8.cshtml https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/src/Platform/BenchmarkApplication.Fortunes.cs#L25 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [3] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/src/Platform/BenchmarkApplication.cs https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
      - dahauns 2y ago
        Thanks a lot for the input, that's quite enlightening - seems like I have been browsing everything but the "Platform" target... Puh, yeah, I see what you mean, much the stuff in [2] and [3] is rather...bespoke, especially compared to the minimal and mvc targets. Not really what I'd consider "realistic" as per the benchmark's definition. But TBH, I wouldn't consider [1] gross, on the contrary - simple, fast, lightweight Razor templating without other MVC (or other external) dependencies isn't that unusual a use case and something I've often thought ASP.NET Core was missing (even Razor Pages feel like overkill if you just want to quickly generate some dynamic HTML).
vips7L 2y ago
Most the Debian benchmarks for C# are cheaty too. They frequently just call out to C libraries rather than use something implemented in the language.
- neonsunset 2y ago
  Not sure which benchmarks you have in mind. Could you provide a link to any of those? .NET's standard library never calls into anything C aside from kernel APIs and certain runtime helpers which is a given. If you meant BenchmarksGame, then it's the other way around - Java is most competitive where it relies heavily on GC[0], and loses in other areas which require capability to write a low-level implementation[1] that C# provides. The only places where there are C calls are pidigts[2] and regex-redux[3] benchmarks, in both of which Java submissions have to import pre-generated or pre-made bindings to GMP and PCRE2 respectively. As do all other languages, with varying degrees of "preparation". [0]: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/binarytrees-java-7.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... [1]: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/spectralnorm-csharpcore-5.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... [2]: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/pidigits-java-3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... [3]: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-java-8.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
  - vips7L 2y ago
    Im sorry, but calling out to C libraries — regardless of the language — is cheating. Just because everyone in the competition is on steroids doesn’t mean you got there legitimately.
    - neonsunset 2y ago
      This is a strange reply given that sibling comment points out it's only 2 out of 10 benchmarks where this is allowed because all languages end up calling out to the same libraries. Even if you prohibit PCRE2, the .NET submissions using out-of-box Regex engine end up being about 4 times faster than Java. Surprisingly, even though .NET's BigInteger is known for its inefficiency, it ends up being more memory efficient and marginally faster at pidigits than a Java submission that does not use GMP. The implementations are not line-by-line equivalent so may not be perfectly representative of performance of each BigInt implementation. My point being - if you look at the submissions closer, the data gives much clearer picture and only supports the argument that C# is a very usable language for solving the tasks one would usually reach for C, C++ or Rust instead.

by EraYaN 2y ago

Ygg2 2y ago

Cheat-y in what way? I don't consider micro benchmarks that interesting especially since C# does have two aces that Java will get in mid term - namely Simd and primitive types. The Tech Empower benchmarks do seem to reflect general state of Java Web Framework ecosystem with Vert.x being they hyper fast web framework and Spring being way slower.

EraYaN 2y ago
Well the amount of tricks that basically nobody can implement in production that those implementations do to get the numbers that they do is ridiculous. Meaning the numbers they get are wildly optimistic at best and misleading at worst. If you take the standard template for any of these frameworks (both Java and C# and any other language) and you add authentication etc, the real performance will be 5-10% of the numbers reported in those benchmarks. Have a look through some of the weirdness in the implementations it's wild (and sometimes educational). The .NET team especially has done stuff specifically to get faster on those benchmarks.
- neonsunset 2y ago
  I second this sentiment. If anything, I genuinely think that the way .NET's TechEmpower submissions look does more damage than good. BenchmarksGame offers a much better close-up comparison by having much simpler submissions that mostly consist of code that you could totally see yourself write in a regular setting. .NET is perfectly capable of standing on its own, and if there are specific areas that need improvement - this should serve as a push to further improve DB driver implementations and make ASP.NET Core more robust against various feature configurations. It is already much, much faster than Spring which is a good start, but it could be pushed further. I'd like to note that neither Go nor Java are viable for high-performance programming in a way that C# is. Neither gives you the required low-level access, performance oriented APIs, ability to use zero-cost abstractions and platform control you get to have in .NET. You can get far with both, but not C++/Rust-far the way you can with C#.
  - Ygg2 2y ago
    > BenchmarksGame offers a much better close-up comparison by having much simpler submissions that mostly consist of code that you could totally see yourself write in a regular setting Yeah, except if you are working on Web servers the quality of the framework and its supporting libraries is much more important than what code could theoretically achieve. What is the point of being able to 200 mph when you only ever drive up to 30mph. > Neither gives you the required low-level access, performance oriented APIs, ability to use zero-cost abstractions. Java is working on high performance abstractions, see Vector API (Simd) and project Valhalla (custom primitive types). Sure C# has a theoretical leg up (for which it paid dearly by causing backwards incompatibility with reified generics) but most of the libraries don't use low-level access or SIMD optimizations or what not.
- dahauns 2y ago
  > Have a look through some of the weirdness in the implementations it's wild (and sometimes educational). The .NET team especially has done stuff specifically to get faster on those benchmarks. Could you give me a pointer or two? I wondered about that myself, especially considering the massive improvement from "old" .NET to the core/kestrel based solutions - but a quick browsing a while ago mostly left me astonished how...well, for lack of a better word, banal most of the code was. Agreed though, lack of all kinds of layers like auth, orm etc. are sadly a drawback of these kinds of benchmarks, if understandable - it would make comparability even trickier and has the danger of the comparison matrix of systems/frameworks/libraries exploding in size. But yeah, would be nice datapoints to have. :)
  - EraYaN 2y ago
    They don't even use Razor Pages but a custom RazorSlices package to do the templating [1]. Yes, that is much faster because it removes MVC and a ton of infrastructure but it's also kind of gross. Also the use of stuff like UnsafePreferInlineScheduling has some downsides (running application code on the IO thread) and honestly I'd never use in production. The custom BufferWriter stuff is pretty neat though, although also not really something most people will reach for. And there is more, like the caching of StringBuilders etc. But it also doesn't use the actual HTTP server to build headers, but they just dump a string into the socket [2], feels a bit unrealistic to me. In general the BenchmarkApplication class [3] is full of non-standard stuff that you'd normally let the framework handle. [1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/src/Platform/Templates/FortunesUtf8.cshtml https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/src/Platform/BenchmarkApplication.Fortunes.cs#L25 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [3] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/src/Platform/BenchmarkApplication.cs https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
    - dahauns 2y ago
      Thanks a lot for the input, that's quite enlightening - seems like I have been browsing everything but the "Platform" target... Puh, yeah, I see what you mean, much the stuff in [2] and [3] is rather...bespoke, especially compared to the minimal and mvc targets. Not really what I'd consider "realistic" as per the benchmark's definition. But TBH, I wouldn't consider [1] gross, on the contrary - simple, fast, lightweight Razor templating without other MVC (or other external) dependencies isn't that unusual a use case and something I've often thought ASP.NET Core was missing (even Razor Pages feel like overkill if you just want to quickly generate some dynamic HTML).

vips7L 2y ago

Most the Debian benchmarks for C# are cheaty too. They frequently just call out to C libraries rather than use something implemented in the language.

neonsunset 2y ago
Not sure which benchmarks you have in mind. Could you provide a link to any of those? .NET's standard library never calls into anything C aside from kernel APIs and certain runtime helpers which is a given. If you meant BenchmarksGame, then it's the other way around - Java is most competitive where it relies heavily on GC[0], and loses in other areas which require capability to write a low-level implementation[1] that C# provides. The only places where there are C calls are pidigts[2] and regex-redux[3] benchmarks, in both of which Java submissions have to import pre-generated or pre-made bindings to GMP and PCRE2 respectively. As do all other languages, with varying degrees of "preparation". [0]: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/binarytrees-java-7.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... [1]: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/spectralnorm-csharpcore-5.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... [2]: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/pidigits-java-3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... [3]: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-java-8.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- vips7L 2y ago
  Im sorry, but calling out to C libraries — regardless of the language — is cheating. Just because everyone in the competition is on steroids doesn’t mean you got there legitimately.
  - neonsunset 2y ago
    This is a strange reply given that sibling comment points out it's only 2 out of 10 benchmarks where this is allowed because all languages end up calling out to the same libraries. Even if you prohibit PCRE2, the .NET submissions using out-of-box Regex engine end up being about 4 times faster than Java. Surprisingly, even though .NET's BigInteger is known for its inefficiency, it ends up being more memory efficient and marginally faster at pidigits than a Java submission that does not use GMP. The implementations are not line-by-line equivalent so may not be perfectly representative of performance of each BigInt implementation. My point being - if you look at the submissions closer, the data gives much clearer picture and only supports the argument that C# is a very usable language for solving the tasks one would usually reach for C, C++ or Rust instead.