3 ms·

A significant part of it is that what engineers believe was effectively true at one time. They simply haven't revisited those beliefs or verified their relevanc

A significant part of it is that what engineers believe was effectively true at one time. They simply haven't revisited those beliefs or verified their relevance in a long time. It isn't a terrible heuristic for life in general to assume that what worked ten years ago will work today. The rate at which the equilibriums shift due to changes in hardware and software environments when designing for system performance is so rapid that you need to make a continuous habit of checking that your understanding of how the world works maps to reality. I've solved a lot of arguments with godbolt and simple performance tests. Some topics are recurring themes among software engineers e.g.: - compilers are almost always better at micro-optimizations than you are - disk I/O is almost never a bottleneck in competent designs - brute-force sequential scans are often optimal algorithms - memory is best treated as a block device - vectorization can offer large performance gains - etc... No one is immune to this. I am sometimes surprised at the extent to which assumptions are no longer true when I revisit optimization work I did 10+ years ago. Most performance these days is architectural, so getting the initial design right often has a bigger impact than micro-optimizations and localized Big-O tweaks. You can always go back and tweak algorithms or codegen later but architecture is permanent.

neonsunset 3y ago
.NET is a particularly bad case for this because it was a decade of few performance improvements, which caused a certain intuition to develop within the industry, then 6-8 years of significant changes each year (with most wins compressed to the last 4 years or so). Companies moving from .NET Framework 4.6/7/8 to .NET 8 experience a 10x average performance improvement, which naturally comes with rendering a lot of performance know-how obsolete overnight. (the techniques that used to work were similar to earlier Java versions and overall very dynamic languages with some exceptions, the techniques that still work and now are required today are the same as in C++ or Rust)
- throwaway2037 3y ago
  .NET 4.6 to .NET 8 is a 10x "average" performance improvement. I find this hard to believe. In what scenarios? I tried to Google for it and found very little hard evidence.
  - neonsunset 3y ago
    In general purpose scenarios, particularly in codebases which have high amount of abstractions, use ASP.NET Core and EF Core, parse and de/serialize text with the use of JSON, Regex and other options, have network and file IO, and are deployed on many-core hosts/container images. There are a few articles on msft devblogs that cover from-netframework migration to older versions (Core 3.1, 5/6/7): - https://devblogs.microsoft.com/dotnet/bing-ads-campaign-platform-journey-to-dotnet-6/ https://devblogs.microsoft.com/dotnet/bing-ads-campaign-plat... - https://devblogs.microsoft.com/dotnet/microsoft-graph-dotnet-6-journey/ https://devblogs.microsoft.com/dotnet/microsoft-graph-dotnet... - https://devblogs.microsoft.com/dotnet/the-azure-cosmos-db-journey-to-net-6/ https://devblogs.microsoft.com/dotnet/the-azure-cosmos-db-jo... - https://devblogs.microsoft.com/dotnet/one-service-journey-to-dotnet-6/ https://devblogs.microsoft.com/dotnet/one-service-journey-to... - https://devblogs.microsoft.com/dotnet/microsoft-commerce-dotnet-6-migration-journey/ https://devblogs.microsoft.com/dotnet/microsoft-commerce-dot... The tl;dr is depending on codebase the latency reduction was anywhere from 2x to 6x, varying per percentile, or the RPS was maintained with CPU usage dropping by ~2-6x. Now, these are codebases of likely above average quality. If you consider that moving 6 -> 8 yields another up to 15-30% on average through improved and enabled by default DynamicPGO, and if you also consider that the average codebase is of worse quality than whatever msft has, meaning that DPGO-reliant optimizations scale way better, it is not difficult to see the 10x number. Keep in mind that while particular regular piece of enterprise code could have improved within bounds of "poor netfx codegen" -> "not far from LLVM with FLTO and PGO", the bottlenecks have changed significantly where previously they could have been in lock contention (within GC or user code), object allocation, object memory copying, e.g. for financial domains - anything including possibly complex Regex queries on imported payment reports (these alone have now difference anywhere between 2 and >1000[0]), and for pretty much every code base also in interface/virtual dispatch for layers upon layers of "clean architecture" solutions. The vast majority of performance improvements (both compiler+gc and CoreLib+frameworks), which is difficult to think about, given it was 8 years, address the above first and foremost. At my previous employer the migration from NETFX 4.6 to .NET Core 3.1, while also deploying to much more constrained container images compared to beefy Windows Server hosts, reduced latency of most requests by the same factor of >5x (certain request type went from 2s to 350ms). It was my first wow moment when I decided to stay with .NET rather than move over to Go back then (was never a fan of syntax though, and other issues, which subsequently got fixed in .NET, that Go still has, are not tolerable for me). [0] Cumulative of https://devblogs.microsoft.com/dotnet/regex-performance-improvements-in-net-5/ https://devblogs.microsoft.com/dotnet/regex-performance-impr... https://devblogs.microsoft.com/dotnet/regular-expression-improvements-in-dotnet-7/ https://devblogs.microsoft.com/dotnet/regular-expression-imp... https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-8/#regex https://devblogs.microsoft.com/dotnet/performance-improvemen...
tombert 3y ago
Yep, completely agree with you on this. Intuition is often wrong, or at least outdated. When I'm building stuff I try my best to focus on "correctness", and try to come up with an algorithm/design that will encompass all realistic use cases. If I focus on that, it's relatively easy to go back and convert my `decimal` type to a float64, or even convert an if statement into a switch if it's actually faster.

neonsunset 3y ago

.NET is a particularly bad case for this because it was a decade of few performance improvements, which caused a certain intuition to develop within the industry, then 6-8 years of significant changes each year (with most wins compressed to the last 4 years or so). Companies moving from .NET Framework 4.6/7/8 to .NET 8 experience a 10x average performance improvement, which naturally comes with rendering a lot of performance know-how obsolete overnight. (the techniques that used to work were similar to earlier Java versions and overall very dynamic languages with some exceptions, the techniques that still work and now are required today are the same as in C++ or Rust)

throwaway2037 3y ago
.NET 4.6 to .NET 8 is a 10x "average" performance improvement. I find this hard to believe. In what scenarios? I tried to Google for it and found very little hard evidence.
- neonsunset 3y ago
  In general purpose scenarios, particularly in codebases which have high amount of abstractions, use ASP.NET Core and EF Core, parse and de/serialize text with the use of JSON, Regex and other options, have network and file IO, and are deployed on many-core hosts/container images. There are a few articles on msft devblogs that cover from-netframework migration to older versions (Core 3.1, 5/6/7): - https://devblogs.microsoft.com/dotnet/bing-ads-campaign-platform-journey-to-dotnet-6/ https://devblogs.microsoft.com/dotnet/bing-ads-campaign-plat... - https://devblogs.microsoft.com/dotnet/microsoft-graph-dotnet-6-journey/ https://devblogs.microsoft.com/dotnet/microsoft-graph-dotnet... - https://devblogs.microsoft.com/dotnet/the-azure-cosmos-db-journey-to-net-6/ https://devblogs.microsoft.com/dotnet/the-azure-cosmos-db-jo... - https://devblogs.microsoft.com/dotnet/one-service-journey-to-dotnet-6/ https://devblogs.microsoft.com/dotnet/one-service-journey-to... - https://devblogs.microsoft.com/dotnet/microsoft-commerce-dotnet-6-migration-journey/ https://devblogs.microsoft.com/dotnet/microsoft-commerce-dot... The tl;dr is depending on codebase the latency reduction was anywhere from 2x to 6x, varying per percentile, or the RPS was maintained with CPU usage dropping by ~2-6x. Now, these are codebases of likely above average quality. If you consider that moving 6 -> 8 yields another up to 15-30% on average through improved and enabled by default DynamicPGO, and if you also consider that the average codebase is of worse quality than whatever msft has, meaning that DPGO-reliant optimizations scale way better, it is not difficult to see the 10x number. Keep in mind that while particular regular piece of enterprise code could have improved within bounds of "poor netfx codegen" -> "not far from LLVM with FLTO and PGO", the bottlenecks have changed significantly where previously they could have been in lock contention (within GC or user code), object allocation, object memory copying, e.g. for financial domains - anything including possibly complex Regex queries on imported payment reports (these alone have now difference anywhere between 2 and >1000[0]), and for pretty much every code base also in interface/virtual dispatch for layers upon layers of "clean architecture" solutions. The vast majority of performance improvements (both compiler+gc and CoreLib+frameworks), which is difficult to think about, given it was 8 years, address the above first and foremost. At my previous employer the migration from NETFX 4.6 to .NET Core 3.1, while also deploying to much more constrained container images compared to beefy Windows Server hosts, reduced latency of most requests by the same factor of >5x (certain request type went from 2s to 350ms). It was my first wow moment when I decided to stay with .NET rather than move over to Go back then (was never a fan of syntax though, and other issues, which subsequently got fixed in .NET, that Go still has, are not tolerable for me). [0] Cumulative of https://devblogs.microsoft.com/dotnet/regex-performance-improvements-in-net-5/ https://devblogs.microsoft.com/dotnet/regex-performance-impr... https://devblogs.microsoft.com/dotnet/regular-expression-improvements-in-dotnet-7/ https://devblogs.microsoft.com/dotnet/regular-expression-imp... https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-8/#regex https://devblogs.microsoft.com/dotnet/performance-improvemen...

tombert 3y ago

Yep, completely agree with you on this. Intuition is often wrong, or at least outdated. When I'm building stuff I try my best to focus on "correctness", and try to come up with an algorithm/design that will encompass all realistic use cases. If I focus on that, it's relatively easy to go back and convert my `decimal` type to a float64, or even convert an if statement into a switch if it's actually faster.