4 ms·

.NET is a particularly bad case for this because it was a decade of few performance improvements, which caused a certain intuition to develop within the industr

.NET is a particularly bad case for this because it was a decade of few performance improvements, which caused a certain intuition to develop within the industry, then 6-8 years of significant changes each year (with most wins compressed to the last 4 years or so). Companies moving from .NET Framework 4.6/7/8 to .NET 8 experience a 10x average performance improvement, which naturally comes with rendering a lot of performance know-how obsolete overnight. (the techniques that used to work were similar to earlier Java versions and overall very dynamic languages with some exceptions, the techniques that still work and now are required today are the same as in C++ or Rust)

throwaway2037 3y ago
.NET 4.6 to .NET 8 is a 10x "average" performance improvement. I find this hard to believe. In what scenarios? I tried to Google for it and found very little hard evidence.
- neonsunset 3y ago
  In general purpose scenarios, particularly in codebases which have high amount of abstractions, use ASP.NET Core and EF Core, parse and de/serialize text with the use of JSON, Regex and other options, have network and file IO, and are deployed on many-core hosts/container images. There are a few articles on msft devblogs that cover from-netframework migration to older versions (Core 3.1, 5/6/7): - https://devblogs.microsoft.com/dotnet/bing-ads-campaign-platform-journey-to-dotnet-6/ https://devblogs.microsoft.com/dotnet/bing-ads-campaign-plat... - https://devblogs.microsoft.com/dotnet/microsoft-graph-dotnet-6-journey/ https://devblogs.microsoft.com/dotnet/microsoft-graph-dotnet... - https://devblogs.microsoft.com/dotnet/the-azure-cosmos-db-journey-to-net-6/ https://devblogs.microsoft.com/dotnet/the-azure-cosmos-db-jo... - https://devblogs.microsoft.com/dotnet/one-service-journey-to-dotnet-6/ https://devblogs.microsoft.com/dotnet/one-service-journey-to... - https://devblogs.microsoft.com/dotnet/microsoft-commerce-dotnet-6-migration-journey/ https://devblogs.microsoft.com/dotnet/microsoft-commerce-dot... The tl;dr is depending on codebase the latency reduction was anywhere from 2x to 6x, varying per percentile, or the RPS was maintained with CPU usage dropping by ~2-6x. Now, these are codebases of likely above average quality. If you consider that moving 6 -> 8 yields another up to 15-30% on average through improved and enabled by default DynamicPGO, and if you also consider that the average codebase is of worse quality than whatever msft has, meaning that DPGO-reliant optimizations scale way better, it is not difficult to see the 10x number. Keep in mind that while particular regular piece of enterprise code could have improved within bounds of "poor netfx codegen" -> "not far from LLVM with FLTO and PGO", the bottlenecks have changed significantly where previously they could have been in lock contention (within GC or user code), object allocation, object memory copying, e.g. for financial domains - anything including possibly complex Regex queries on imported payment reports (these alone have now difference anywhere between 2 and >1000[0]), and for pretty much every code base also in interface/virtual dispatch for layers upon layers of "clean architecture" solutions. The vast majority of performance improvements (both compiler+gc and CoreLib+frameworks), which is difficult to think about, given it was 8 years, address the above first and foremost. At my previous employer the migration from NETFX 4.6 to .NET Core 3.1, while also deploying to much more constrained container images compared to beefy Windows Server hosts, reduced latency of most requests by the same factor of >5x (certain request type went from 2s to 350ms). It was my first wow moment when I decided to stay with .NET rather than move over to Go back then (was never a fan of syntax though, and other issues, which subsequently got fixed in .NET, that Go still has, are not tolerable for me). [0] Cumulative of https://devblogs.microsoft.com/dotnet/regex-performance-improvements-in-net-5/ https://devblogs.microsoft.com/dotnet/regex-performance-impr... https://devblogs.microsoft.com/dotnet/regular-expression-improvements-in-dotnet-7/ https://devblogs.microsoft.com/dotnet/regular-expression-imp... https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-8/#regex https://devblogs.microsoft.com/dotnet/performance-improvemen...
  - rerdavies 3y ago
    Cheating. All of the 6x performance improvement cases seem to be related to using the .net based Kestrel web server instead of IIS web server, which requires marshalling and interprocess communication. Several of the 2x gains appear to be related to using a different database backend. Claims that regex performance has improved a thousand-fold.... seem more troubling than cause for celebration. Were you not precompiling your regex's in the older code? That would be a bug. Somewhere in there, there might be 30% improvements in .net codegen (it's hard to tell). Profile Guided Optimization (PGO) seems to provide a 35% performance improvement over older versions of .net with PGO disabled. But that's dishonest. PGO was around long before .net Core. And claiming that PGO will provide 10x performance because our code is worse than Microsoft's code insults both our code and our intelligence.
    - throwaway2037 3y ago
      > Were you not precompiling your regex's in the older code? That would be a bug. I never heard of this before. Perl has legendary fast regexen and I never heard of this feature. Does Java do it? I don't think so, and the regexes are fast enough in my experience. Can you name a language when regexen are precompiled?
    - ygra 3y ago
      Not sure about the 10×, either, and if true it would involve more than just the JIT changes. But changing ASP.NET to ASP.NET Core at the same time and the web server as well as other libraries may make it plausible. For certain applications moving from .NET Framework to .NET isn't so simple when they have dependencies and those have changed their API significantly. And in that case most of the newer stuff seems to be built with performance in mind. So you gain 30 % from the JIT, 2× from Kestrel, and so on. Perhaps. With a Roslyn-based compiler at work I saw 20 % perf improvement just by switching from .NET Core 3.1 to .NET 6. No idea how slow .NET Framework was, though. I probably can't target the code to that anymore. But for regex even with precompilation, the compiler got a lot better at transforming the regex into an equivalent regex that performs better (automatic atomic grouping to reduce unnecessary backtracking when it's statically known that backtracking won't create more matches for example) and it also benefits a lot from the various vectorized implementations of Index of, etc. Typically with each improvement of one of those core methods for searching stuff in memory there's a corresponding change that uses them in regex. So where in .NET Framework a regex might walk through a whole string character by character multiple times with backtracking it might be replaced with effectively an EndsWith and LastIndexOfAny call in newer versions.

throwaway2037 3y ago

.NET 4.6 to .NET 8 is a 10x "average" performance improvement. I find this hard to believe. In what scenarios? I tried to Google for it and found very little hard evidence.

neonsunset 3y ago
In general purpose scenarios, particularly in codebases which have high amount of abstractions, use ASP.NET Core and EF Core, parse and de/serialize text with the use of JSON, Regex and other options, have network and file IO, and are deployed on many-core hosts/container images. There are a few articles on msft devblogs that cover from-netframework migration to older versions (Core 3.1, 5/6/7): - https://devblogs.microsoft.com/dotnet/bing-ads-campaign-platform-journey-to-dotnet-6/ https://devblogs.microsoft.com/dotnet/bing-ads-campaign-plat... - https://devblogs.microsoft.com/dotnet/microsoft-graph-dotnet-6-journey/ https://devblogs.microsoft.com/dotnet/microsoft-graph-dotnet... - https://devblogs.microsoft.com/dotnet/the-azure-cosmos-db-journey-to-net-6/ https://devblogs.microsoft.com/dotnet/the-azure-cosmos-db-jo... - https://devblogs.microsoft.com/dotnet/one-service-journey-to-dotnet-6/ https://devblogs.microsoft.com/dotnet/one-service-journey-to... - https://devblogs.microsoft.com/dotnet/microsoft-commerce-dotnet-6-migration-journey/ https://devblogs.microsoft.com/dotnet/microsoft-commerce-dot... The tl;dr is depending on codebase the latency reduction was anywhere from 2x to 6x, varying per percentile, or the RPS was maintained with CPU usage dropping by ~2-6x. Now, these are codebases of likely above average quality. If you consider that moving 6 -> 8 yields another up to 15-30% on average through improved and enabled by default DynamicPGO, and if you also consider that the average codebase is of worse quality than whatever msft has, meaning that DPGO-reliant optimizations scale way better, it is not difficult to see the 10x number. Keep in mind that while particular regular piece of enterprise code could have improved within bounds of "poor netfx codegen" -> "not far from LLVM with FLTO and PGO", the bottlenecks have changed significantly where previously they could have been in lock contention (within GC or user code), object allocation, object memory copying, e.g. for financial domains - anything including possibly complex Regex queries on imported payment reports (these alone have now difference anywhere between 2 and >1000[0]), and for pretty much every code base also in interface/virtual dispatch for layers upon layers of "clean architecture" solutions. The vast majority of performance improvements (both compiler+gc and CoreLib+frameworks), which is difficult to think about, given it was 8 years, address the above first and foremost. At my previous employer the migration from NETFX 4.6 to .NET Core 3.1, while also deploying to much more constrained container images compared to beefy Windows Server hosts, reduced latency of most requests by the same factor of >5x (certain request type went from 2s to 350ms). It was my first wow moment when I decided to stay with .NET rather than move over to Go back then (was never a fan of syntax though, and other issues, which subsequently got fixed in .NET, that Go still has, are not tolerable for me). [0] Cumulative of https://devblogs.microsoft.com/dotnet/regex-performance-improvements-in-net-5/ https://devblogs.microsoft.com/dotnet/regex-performance-impr... https://devblogs.microsoft.com/dotnet/regular-expression-improvements-in-dotnet-7/ https://devblogs.microsoft.com/dotnet/regular-expression-imp... https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-8/#regex https://devblogs.microsoft.com/dotnet/performance-improvemen...
- rerdavies 3y ago
  Cheating. All of the 6x performance improvement cases seem to be related to using the .net based Kestrel web server instead of IIS web server, which requires marshalling and interprocess communication. Several of the 2x gains appear to be related to using a different database backend. Claims that regex performance has improved a thousand-fold.... seem more troubling than cause for celebration. Were you not precompiling your regex's in the older code? That would be a bug. Somewhere in there, there might be 30% improvements in .net codegen (it's hard to tell). Profile Guided Optimization (PGO) seems to provide a 35% performance improvement over older versions of .net with PGO disabled. But that's dishonest. PGO was around long before .net Core. And claiming that PGO will provide 10x performance because our code is worse than Microsoft's code insults both our code and our intelligence.
  - throwaway2037 3y ago
    > Were you not precompiling your regex's in the older code? That would be a bug. I never heard of this before. Perl has legendary fast regexen and I never heard of this feature. Does Java do it? I don't think so, and the regexes are fast enough in my experience. Can you name a language when regexen are precompiled?
  - ygra 3y ago
    Not sure about the 10×, either, and if true it would involve more than just the JIT changes. But changing ASP.NET to ASP.NET Core at the same time and the web server as well as other libraries may make it plausible. For certain applications moving from .NET Framework to .NET isn't so simple when they have dependencies and those have changed their API significantly. And in that case most of the newer stuff seems to be built with performance in mind. So you gain 30 % from the JIT, 2× from Kestrel, and so on. Perhaps. With a Roslyn-based compiler at work I saw 20 % perf improvement just by switching from .NET Core 3.1 to .NET 6. No idea how slow .NET Framework was, though. I probably can't target the code to that anymore. But for regex even with precompilation, the compiler got a lot better at transforming the regex into an equivalent regex that performs better (automatic atomic grouping to reduce unnecessary backtracking when it's statically known that backtracking won't create more matches for example) and it also benefits a lot from the various vectorized implementations of Index of, etc. Typically with each improvement of one of those core methods for searching stuff in memory there's a corresponding change that uses them in regex. So where in .NET Framework a regex might walk through a whole string character by character multiple times with backtracking it might be replaced with effectively an EndsWith and LastIndexOfAny call in newer versions.