5 ms·

Blog author here with a regretful correction. This result is not accurate, mea culpa. The headline should read: errors.Is() is 500% slower. Basically: the crit

by zachmu 2y ago

Blog author here with a regretful correction. This result is not accurate, mea culpa. The headline should read: errors.Is() is 500% slower. Basically: the critical functions in the benchmark are small enough they were being inlined by the compiler, which means it's possible for the compiler to further optimize the loop to avoid all comparisons in some cases, producing an inaccurate result for some of the benchmarks. You can fix this by adding noinline directives to the methods. I'll be publishing an update to the article and a post-mortem on how this slipped past review. The rank ordering of techniques in the article is unchanged, but the magnitude of the difference is not nearly so large.

jay-barronville 2y ago
Do you notice any difference if you update `BenchmarkFoundErrorsIs()` to the following? func BenchmarkFoundErrorsIs(b *testing.B) { var es errStore for i := 0; i < b.N; i++ { val, err := es.GetValue(true) if err != nil { if errors.Is(err, notFoundErr) { b.Fatal("expected found") } else { b.Fatal(err) } } if val == nil { b.Fatal("expected not nil") } } }
- zachmu 2y ago
  That's the BenchmarkFoundErrorsIsNilCheck variation, and it eliminates the performance penalty on the happy path. Still just as bad on the error path.
  - jay-barronville 2y ago
    My apologies! I missed that you covered that variation too. (Good to know it works as I expected though!)
rsc 2y ago
Thanks for posting the update here. I was going to point out that 0.5ns for anything is almost always a sign of benchmarks being optimized away. But even if those numbers had been accurate, it is important to think about how they compare to the operations being performed. Taking 4 nanoseconds to check for an error sentinel is not a huge deal when you've spent microseconds or even milliseconds waiting for a network service.
- zachmu 2y ago
  You'll never get those 4 nanoseconds of your life back though
  - philosopher1234 2y ago
    4 nanoseconds spent calling high quality functions is 4 nanoseconds well spent
    - neonsunset 2y ago
      4 nanoseconds is how much it takes for unoptimized errors.Is version of this code[0] to execute in C#, instead of 15-17ns in Go. I don't know how long it will take the industry to learn not to use Go for writing databases. [0]: https://gist.github.com/neon-sunset/65a8deeaac745ba159f990c78681f313 https://gist.github.com/neon-sunset/65a8deeaac745ba159f990c7...
      - zachmu 2y ago
        Our database is within spitting distance of MySQL's raw performance and getting faster all the time. Certainly there are some applications where that will not be fast enough. That didn't prevent MySQL from becoming the worlds most widely deployed free database. And yes, postgres is faster and will eventually pass MySQL in deployments, but that's success also has very little to do with its speed.
      - vips7L 2y ago
        Is that AOT or JIT?
        neonsunset 2y ago
        Updated the numbers in the gist with more context. This is .NET 9 preview 5 JIT or .NET 8 + DPGO casts and VTable profiling feature (it was tuned, stabilized and is now enabled by default in 9). This would be representative for a long-running DB workload as the one the original blog post refers to, which wants to use JIT rather than AOT. Practically speaking, the example is implemented in a way to make it logically closer to Golang one. However, splitting the interface check and the cast defeats default behavior of .NET 8 JIT's guarded and ILC's exact devirtualization. Ideally, you would not perform type tests and just express it through a single interface abstraction that inherits from IEquatable<T> - this is what regular code looks like and what the compiler is optimized towards (e.g. if ILC can see that only a single type implements an interface, all callsites referring to it are unconditionally devirtualized). Tl;Dr: CPU Freq: 3.22GHz, single cycle: ~0.31ns L1 reference roundtrip lat.: 4 cycles (iirc for Firestorm) Cost of GetValue -> (object?, IError?) + Errors.Is call -- JIT -- .NET 9: 3.24ns value ret, 3.58ns error ret .NET 8: 5.56ns value ret, 5.86ns error ret -- AOT -- .NET 9: 5.72ns value ret, 5.91ns error ret .NET 8: 5.52ns value ret, 5.89ns error ret Note: .NET 9 numbers are not finalized - it's still 5 months away from release.
zachmu 2y ago
Corrections and new title are now live. Thank you for your forbearance.
hinkley 2y ago
The thing with programming language evolution is that quite often everything except the error handling gets faster over time. If you have an app that makes aggressive use of errors, you’re not going to see the same speedups from version to version that others are seeing. Particularly in JITed languages, if the multiplier is hyperbole today, there may be a day in the future where it’s accurate.
- rafaelmn 2y ago
  > The thing with programming language evolution is that quite often everything except the error handling gets faster over time. Funny I was just reading this the other day : https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotnet-9/runtime https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotn... It's probably not the first thing you spend effort on when optimizing but it gets better over time.
- samatman 2y ago
  The minimum cost of an error is one cold branch. Which is effectively free on the happy path, and as cheap as it possibly can be on the error path. A well-designed error handling system can feasibly reach this goal, at which point no more optimization is possible. Edit @hinkley: I am unfortunately in hacker jail, and have used up my post quota, so I have to edit my post rather than reply to yours. Reply follows. Throwing, the way Java and C++ do it, exceeds the minimum cost of an error, yes. Not that this is some sort of show-stopping problem, I'm discussing the minimum cost, not what level of cost makes errors too expensive. A branch-not-taken, which is the opposite of processing an error, caught or otherwise, is what I described as practically free. Not always. If the only path forward is, for instance, to reference a pointer, and the error would mean that the pointer is a null pointer, the chip would have to wait for the branch to coalesce, which might mean a minor pipeline stall. But if the processor can do further work, such as checking for an error on malformed input, where it's legal to speculatively execute the next decode on the predicted branch, it really is practically free. It costs considerably less than one instruction. If the branch triggers, the processor spills all the speculative work and switches. Since errors are expected less frequently than success, more or less by definition, that's the correct tradeoff. This is why I described the error branch as "cold". The processor should predict and speculate on the happy path. Zig is an example of a language where errors can be as inexpensive as a superscalar check on a given register while the main path moves forward. This is also possible in C when checking errno(). I can't speak for Rust, but enormous amounts of work have gone into that compiler, and it should indeed be possible to compile a match-on-Result to the same efficient level. Perhaps trickier since Rust doesn't "know" what an error is, but I suspect that in practice it can get this right. Not that it's even mostly up to the compiler: there are ASM hints for the branch predictor in some instruction sets, but modern branch predictors don't need those to handle the case where 99+% of the time execution proceeds on one branch. It's conventional for compilers to emit "branch on cold, proceed on hot", but it's been a long time since that has mattered for execution speed. You'll get a bit tighter on a chip with no predictor and a shallow pipeline, like AVR, but for anything which can run an operating system it's almost irrelevant now. I see no a priori reason why Go shouldn't achieve the minimum level either. Languages which use exceptions and try/catch must do some bookkeeping, which imposes a cost on the code whether an exception is raised, or not. That can be minimized, but not to the limit I'm talking about. You might find this link interesting: https://ziglang.org/documentation/0.12.0/#Implementation-Details https://ziglang.org/documentation/0.12.0/#Implementation-Det... Note that the costs described are for the ability to generate an error trace, not error handling itself. This cost is only paid in debug and ReleaseSafe modes.

by zachmu 2y ago

jay-barronville 2y ago

Do you notice any difference if you update `BenchmarkFoundErrorsIs()` to the following? func BenchmarkFoundErrorsIs(b *testing.B) { var es errStore for i := 0; i < b.N; i++ { val, err := es.GetValue(true) if err != nil { if errors.Is(err, notFoundErr) { b.Fatal("expected found") } else { b.Fatal(err) } } if val == nil { b.Fatal("expected not nil") } } }

zachmu 2y ago
That's the BenchmarkFoundErrorsIsNilCheck variation, and it eliminates the performance penalty on the happy path. Still just as bad on the error path.
- jay-barronville 2y ago
  My apologies! I missed that you covered that variation too. (Good to know it works as I expected though!)

rsc 2y ago

Thanks for posting the update here. I was going to point out that 0.5ns for anything is almost always a sign of benchmarks being optimized away. But even if those numbers had been accurate, it is important to think about how they compare to the operations being performed. Taking 4 nanoseconds to check for an error sentinel is not a huge deal when you've spent microseconds or even milliseconds waiting for a network service.

zachmu 2y ago
You'll never get those 4 nanoseconds of your life back though
- philosopher1234 2y ago
  4 nanoseconds spent calling high quality functions is 4 nanoseconds well spent
  - neonsunset 2y ago
    4 nanoseconds is how much it takes for unoptimized errors.Is version of this code[0] to execute in C#, instead of 15-17ns in Go. I don't know how long it will take the industry to learn not to use Go for writing databases. [0]: https://gist.github.com/neon-sunset/65a8deeaac745ba159f990c78681f313 https://gist.github.com/neon-sunset/65a8deeaac745ba159f990c7...
    - zachmu 2y ago
      Our database is within spitting distance of MySQL's raw performance and getting faster all the time. Certainly there are some applications where that will not be fast enough. That didn't prevent MySQL from becoming the worlds most widely deployed free database. And yes, postgres is faster and will eventually pass MySQL in deployments, but that's success also has very little to do with its speed.
    - vips7L 2y ago
      Is that AOT or JIT?
      - neonsunset 2y ago
        Updated the numbers in the gist with more context. This is .NET 9 preview 5 JIT or .NET 8 + DPGO casts and VTable profiling feature (it was tuned, stabilized and is now enabled by default in 9). This would be representative for a long-running DB workload as the one the original blog post refers to, which wants to use JIT rather than AOT. Practically speaking, the example is implemented in a way to make it logically closer to Golang one. However, splitting the interface check and the cast defeats default behavior of .NET 8 JIT's guarded and ILC's exact devirtualization. Ideally, you would not perform type tests and just express it through a single interface abstraction that inherits from IEquatable<T> - this is what regular code looks like and what the compiler is optimized towards (e.g. if ILC can see that only a single type implements an interface, all callsites referring to it are unconditionally devirtualized). Tl;Dr: CPU Freq: 3.22GHz, single cycle: ~0.31ns L1 reference roundtrip lat.: 4 cycles (iirc for Firestorm) Cost of GetValue -> (object?, IError?) + Errors.Is call -- JIT -- .NET 9: 3.24ns value ret, 3.58ns error ret .NET 8: 5.56ns value ret, 5.86ns error ret -- AOT -- .NET 9: 5.72ns value ret, 5.91ns error ret .NET 8: 5.52ns value ret, 5.89ns error ret Note: .NET 9 numbers are not finalized - it's still 5 months away from release.

zachmu 2y ago

Corrections and new title are now live. Thank you for your forbearance.

hinkley 2y ago

The thing with programming language evolution is that quite often everything except the error handling gets faster over time. If you have an app that makes aggressive use of errors, you’re not going to see the same speedups from version to version that others are seeing. Particularly in JITed languages, if the multiplier is hyperbole today, there may be a day in the future where it’s accurate.

rafaelmn 2y ago
> The thing with programming language evolution is that quite often everything except the error handling gets faster over time. Funny I was just reading this the other day : https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotnet-9/runtime https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotn... It's probably not the first thing you spend effort on when optimizing but it gets better over time.
samatman 2y ago
The minimum cost of an error is one cold branch. Which is effectively free on the happy path, and as cheap as it possibly can be on the error path. A well-designed error handling system can feasibly reach this goal, at which point no more optimization is possible. Edit @hinkley: I am unfortunately in hacker jail, and have used up my post quota, so I have to edit my post rather than reply to yours. Reply follows. Throwing, the way Java and C++ do it, exceeds the minimum cost of an error, yes. Not that this is some sort of show-stopping problem, I'm discussing the minimum cost, not what level of cost makes errors too expensive. A branch-not-taken, which is the opposite of processing an error, caught or otherwise, is what I described as practically free. Not always. If the only path forward is, for instance, to reference a pointer, and the error would mean that the pointer is a null pointer, the chip would have to wait for the branch to coalesce, which might mean a minor pipeline stall. But if the processor can do further work, such as checking for an error on malformed input, where it's legal to speculatively execute the next decode on the predicted branch, it really is practically free. It costs considerably less than one instruction. If the branch triggers, the processor spills all the speculative work and switches. Since errors are expected less frequently than success, more or less by definition, that's the correct tradeoff. This is why I described the error branch as "cold". The processor should predict and speculate on the happy path. Zig is an example of a language where errors can be as inexpensive as a superscalar check on a given register while the main path moves forward. This is also possible in C when checking errno(). I can't speak for Rust, but enormous amounts of work have gone into that compiler, and it should indeed be possible to compile a match-on-Result to the same efficient level. Perhaps trickier since Rust doesn't "know" what an error is, but I suspect that in practice it can get this right. Not that it's even mostly up to the compiler: there are ASM hints for the branch predictor in some instruction sets, but modern branch predictors don't need those to handle the case where 99+% of the time execution proceeds on one branch. It's conventional for compilers to emit "branch on cold, proceed on hot", but it's been a long time since that has mattered for execution speed. You'll get a bit tighter on a chip with no predictor and a shallow pipeline, like AVR, but for anything which can run an operating system it's almost irrelevant now. I see no a priori reason why Go shouldn't achieve the minimum level either. Languages which use exceptions and try/catch must do some bookkeeping, which imposes a cost on the code whether an exception is raised, or not. That can be minimized, but not to the limit I'm talking about. You might find this link interesting: https://ziglang.org/documentation/0.12.0/#Implementation-Details https://ziglang.org/documentation/0.12.0/#Implementation-Det... Note that the costs described are for the ability to generate an error trace, not error handling itself. This cost is only paid in debug and ReleaseSafe modes.