4 ms·

Cheating. All of the 6x performance improvement cases seem to be related to using the .net based Kestrel web server instead of IIS web server, which requires m

Cheating. All of the 6x performance improvement cases seem to be related to using the .net based Kestrel web server instead of IIS web server, which requires marshalling and interprocess communication. Several of the 2x gains appear to be related to using a different database backend. Claims that regex performance has improved a thousand-fold.... seem more troubling than cause for celebration. Were you not precompiling your regex's in the older code? That would be a bug. Somewhere in there, there might be 30% improvements in .net codegen (it's hard to tell). Profile Guided Optimization (PGO) seems to provide a 35% performance improvement over older versions of .net with PGO disabled. But that's dishonest. PGO was around long before .net Core. And claiming that PGO will provide 10x performance because our code is worse than Microsoft's code insults both our code and our intelligence.

throwaway2037 3y ago
> Were you not precompiling your regex's in the older code? That would be a bug. I never heard of this before. Perl has legendary fast regexen and I never heard of this feature. Does Java do it? I don't think so, and the regexes are fast enough in my experience. Can you name a language when regexen are precompiled?
ygra 3y ago
Not sure about the 10×, either, and if true it would involve more than just the JIT changes. But changing ASP.NET to ASP.NET Core at the same time and the web server as well as other libraries may make it plausible. For certain applications moving from .NET Framework to .NET isn't so simple when they have dependencies and those have changed their API significantly. And in that case most of the newer stuff seems to be built with performance in mind. So you gain 30 % from the JIT, 2× from Kestrel, and so on. Perhaps. With a Roslyn-based compiler at work I saw 20 % perf improvement just by switching from .NET Core 3.1 to .NET 6. No idea how slow .NET Framework was, though. I probably can't target the code to that anymore. But for regex even with precompilation, the compiler got a lot better at transforming the regex into an equivalent regex that performs better (automatic atomic grouping to reduce unnecessary backtracking when it's statically known that backtracking won't create more matches for example) and it also benefits a lot from the various vectorized implementations of Index of, etc. Typically with each improvement of one of those core methods for searching stuff in memory there's a corresponding change that uses them in regex. So where in .NET Framework a regex might walk through a whole string character by character multiple times with backtracking it might be replaced with effectively an EndsWith and LastIndexOfAny call in newer versions.
- neonsunset 3y ago
  Roslyn didn't have much of changes in terms of optimizations - it compiles C# to IL so does very little of that, save for switches and certain new or otherwise features like collection literals. You are probably talking about RyuJIT, also called just JIT nowadays :D (the distinction becomes important for targets serviced by Mono, so to outline the difference Mono is usually specified, while CoreCLR and RyuJIT may not be, it also doesn't help that JIT, that is, the IL to machine code compiler, also services NativeAOT, so it gets more annoying to be accurate in a conversation without saying the generic ".net compiler", some people refer to it as JIT/ILC)
  - ygra 3y ago
    No, I meant that we've written a compiler, based on Roslyn, whose runtime for compiling the code has improved by 20 % when switching to .NET 6. And indeed, on the C# -> IL side there's little that's being actually optimized. Besides collection literals there's also switch statements/expressions over strings, along with certain pattern matching constructs that get improved on that side.
    - neonsunset 3y ago
      Interesting! (I was way off the mark, not reading carefully, ha) Is it a public project?
      - ygra 3y ago
        Nope, completely internal and part of how we offer essentially the same product on multiple platforms with minimal integration work. And existing C# → anything compilers are typically too focused on compiling a whole application instead of offering a library with a stable and usable API on the other end, so we had to roll our own.
neonsunset 3y ago
No. DynamicPGO was first introduced in .NET 6 but was not mature and needed two releases worth of work to become enabled by default. It needs no user input and is similar to what OpenJDK Hotspot has been doing for some time and then a little more. It also is required for major features that were strictly not available previously: guarded devirtualization of virtual and interface calls and delegate inlining. Also, IIS hosting through Http.sys is still an option that sees separate set of improvements, but that's not relevant in most situations given the move to .NET 8 from Framework usually also involves replacing Windows Server host with a Linux container (though it works perfectly fine on Windows as well). On Regex, compiled and now source generated automata has seen a lot of work in all recent releases, it is night and day to what it was before - just read the articles. Previously linear scans against heavy internal data structures (matching by hashset) and heavy transient allocations got replaced with bloom-filter style SIMD search and other state of the art text search algorithms[0], on a completely opposite end of a performance spectrum. So when you have compiler improvements multiplied by changes to CoreLib internals multiplied by changes to frameworks built on top - it's achievable with relative ease. .NET Framework, while performing adequately, was still that slow compared to what we got today. [0] https://github.com/dotnet/runtime/tree/main/src/libraries/System.Private.CoreLib/src/System/SearchValues/Strings/Helpers https://github.com/dotnet/runtime/tree/main/src/libraries/Sy...
- rerdavies 3y ago
  Sure. But static PGO was introduced in .Net Framework 4.7.0. And we're talking about apps in production, so there's no excuse NOT to use static PGO on the .net framework 4.7.0 version. And you have misrepresented the contents of the blogs. The projects discussed in the blogs are typically claiming ~30% improvements (perhaps because they weren't using static PGO in their 4.7.0 incarnation), with two dramatic outliers that seem to be related to migrating from IIS to Kestrel.
  - neonsunset 3y ago
    It’s a moot point. Almost no one used static PGO and its feature set was way more limited - it did not have devirtualization which provides the biggest wins. Though you are welcome to disagree it won’t change the reality of the impact .NET 8 release had on real world code. It’s also convenient to ignore the rest of the content at the links but it seems you’re more interested in proving your argument so the data I provided doesn’t matter.
andyayers 3y ago
Something closer to a "pure codegen/runtime" example perhaps: I have data showing Roslyn (the C# compiler, itself written in C#) speeds up between ~2x and ~3x running on .NET 8 vs .NET 4.7.1. Roslyn is built so that it can run either against full framework or core, so it's largely the same application IL.

throwaway2037 3y ago

> Were you not precompiling your regex's in the older code? That would be a bug. I never heard of this before. Perl has legendary fast regexen and I never heard of this feature. Does Java do it? I don't think so, and the regexes are fast enough in my experience. Can you name a language when regexen are precompiled?

ygra 3y ago

Not sure about the 10×, either, and if true it would involve more than just the JIT changes. But changing ASP.NET to ASP.NET Core at the same time and the web server as well as other libraries may make it plausible. For certain applications moving from .NET Framework to .NET isn't so simple when they have dependencies and those have changed their API significantly. And in that case most of the newer stuff seems to be built with performance in mind. So you gain 30 % from the JIT, 2× from Kestrel, and so on. Perhaps. With a Roslyn-based compiler at work I saw 20 % perf improvement just by switching from .NET Core 3.1 to .NET 6. No idea how slow .NET Framework was, though. I probably can't target the code to that anymore. But for regex even with precompilation, the compiler got a lot better at transforming the regex into an equivalent regex that performs better (automatic atomic grouping to reduce unnecessary backtracking when it's statically known that backtracking won't create more matches for example) and it also benefits a lot from the various vectorized implementations of Index of, etc. Typically with each improvement of one of those core methods for searching stuff in memory there's a corresponding change that uses them in regex. So where in .NET Framework a regex might walk through a whole string character by character multiple times with backtracking it might be replaced with effectively an EndsWith and LastIndexOfAny call in newer versions.

neonsunset 3y ago
Roslyn didn't have much of changes in terms of optimizations - it compiles C# to IL so does very little of that, save for switches and certain new or otherwise features like collection literals. You are probably talking about RyuJIT, also called just JIT nowadays :D (the distinction becomes important for targets serviced by Mono, so to outline the difference Mono is usually specified, while CoreCLR and RyuJIT may not be, it also doesn't help that JIT, that is, the IL to machine code compiler, also services NativeAOT, so it gets more annoying to be accurate in a conversation without saying the generic ".net compiler", some people refer to it as JIT/ILC)
- ygra 3y ago
  No, I meant that we've written a compiler, based on Roslyn, whose runtime for compiling the code has improved by 20 % when switching to .NET 6. And indeed, on the C# -> IL side there's little that's being actually optimized. Besides collection literals there's also switch statements/expressions over strings, along with certain pattern matching constructs that get improved on that side.
  - neonsunset 3y ago
    Interesting! (I was way off the mark, not reading carefully, ha) Is it a public project?
    - ygra 3y ago
      Nope, completely internal and part of how we offer essentially the same product on multiple platforms with minimal integration work. And existing C# → anything compilers are typically too focused on compiling a whole application instead of offering a library with a stable and usable API on the other end, so we had to roll our own.

neonsunset 3y ago

No. DynamicPGO was first introduced in .NET 6 but was not mature and needed two releases worth of work to become enabled by default. It needs no user input and is similar to what OpenJDK Hotspot has been doing for some time and then a little more. It also is required for major features that were strictly not available previously: guarded devirtualization of virtual and interface calls and delegate inlining. Also, IIS hosting through Http.sys is still an option that sees separate set of improvements, but that's not relevant in most situations given the move to .NET 8 from Framework usually also involves replacing Windows Server host with a Linux container (though it works perfectly fine on Windows as well). On Regex, compiled and now source generated automata has seen a lot of work in all recent releases, it is night and day to what it was before - just read the articles. Previously linear scans against heavy internal data structures (matching by hashset) and heavy transient allocations got replaced with bloom-filter style SIMD search and other state of the art text search algorithms[0], on a completely opposite end of a performance spectrum. So when you have compiler improvements multiplied by changes to CoreLib internals multiplied by changes to frameworks built on top - it's achievable with relative ease. .NET Framework, while performing adequately, was still that slow compared to what we got today. [0] https://github.com/dotnet/runtime/tree/main/src/libraries/System.Private.CoreLib/src/System/SearchValues/Strings/Helpers https://github.com/dotnet/runtime/tree/main/src/libraries/Sy...

rerdavies 3y ago
Sure. But static PGO was introduced in .Net Framework 4.7.0. And we're talking about apps in production, so there's no excuse NOT to use static PGO on the .net framework 4.7.0 version. And you have misrepresented the contents of the blogs. The projects discussed in the blogs are typically claiming ~30% improvements (perhaps because they weren't using static PGO in their 4.7.0 incarnation), with two dramatic outliers that seem to be related to migrating from IIS to Kestrel.
- neonsunset 3y ago
  It’s a moot point. Almost no one used static PGO and its feature set was way more limited - it did not have devirtualization which provides the biggest wins. Though you are welcome to disagree it won’t change the reality of the impact .NET 8 release had on real world code. It’s also convenient to ignore the rest of the content at the links but it seems you’re more interested in proving your argument so the data I provided doesn’t matter.

andyayers 3y ago

Something closer to a "pure codegen/runtime" example perhaps: I have data showing Roslyn (the C# compiler, itself written in C#) speeds up between ~2x and ~3x running on .NET 8 vs .NET 4.7.1. Roslyn is built so that it can run either against full framework or core, so it's largely the same application IL.