3 ms·

> I’m in the same ballpark as Alexander Yastrebov’s Go version. His solution looks similar to mine: break the file into chunks, use a custom hash table (he even

> I’m in the same ballpark as Alexander Yastrebov’s Go version. His solution looks similar to mine: break the file into chunks, use a custom hash table (he even uses FNV hashing), and parse temperatures as integers. However, he uses memory-mapped files, which I’d ruled out for portability reasons – I’m guessing that’s why his is a bit faster. I am curious, can it be made even faster than this?

makotech221 3y ago
Dunno about Go, but most c# solutions are around 2s and under https://hotforknowledge.com/2024/01/13/1brc-in-dotnet-among-fastest-on-linux-my-optimization-journey/#results https://hotforknowledge.com/2024/01/13/1brc-in-dotnet-among-...
- junto 3y ago
  Wow that’s pretty damn fast. C# has made some improvements in the past years or they have some other advantages?
  - pjmlp 3y ago
    Yes to both. .NET team has been doubling down on performance improvements, people forget CLR also has features to support C like languages (hence Managed C++ and C++/CLI), and many of those capabilities are now surfaced into C# as well.
  - Anon4Now 3y ago
    Stephen Toub wrote a book-length blog post about all the performance improvements made in .NET 8 [1]. Add the option to compile to native. [1] https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-8/ https://devblogs.microsoft.com/dotnet/performance-improvemen...
    - pjmlp 3y ago
      His blogs posts go back all the way to .NET 5, for those curious to some deep dive on performance improvements done on each release.
  - to11mtm 3y ago
    Yeah. A few highlights, as can be seen in some of the blog posts mentioned by other replies: - 'Span<T>' to represent chunks of either managed OR unmanaged memory without using unsafe pointers throughout [0][1] - Not relevant to this task necessarily, but a lot of machinery has been added to allow reuse of objects for tasks like queuing thread pool work, or waiting for an asynchronous result. - Lots of intrinsics helpers for SIMD workloads, and increased usage of such intrinsics in internal parsers/etc. - Generally improving a lot of the internal IO to take advantage of other improvements in the runtime. - PGO (Performance Guided Optimization) on the JIT side, essentially helps with things like better devirt [2] and other improvements. - AOT compilation, if that's your thing, (I do believe the fastest C# 1BRC submissions use this) [0] - To be clear, unsafe can still be faster, however for most cases Span is fine and gives you a little more runtime safety. [1] - You can also grab a Span<T> of a primitive (i.e. int, char) within a method, so long as you don't blow up stack, this is very nice when you need a small buffer for parsing but don't want to thrash the GC or deal with volatile or locks on some sort of pool. [2] - Devirt historically was a problem in 'call heavy' .NET apps when Interfaces are used, before PGO there was more than one library I worked on where we intentionally used abstract base classes rather than interfaces due to the need to squeeze as much out as we could.
krallja 3y ago
There's a layer of pointer indirection when using slices in Go, you may be able to eke out some time by moving to arrays on the stack.
pram 3y ago
From looking at the final code it’s probably the performance of copy() as the biggest hurdle.
scotty79 3y ago
Instead of hash table I'd try sort of "eager" trie inside stack allocated memory. So I can find the slot for the stats of given station after parsing minimal number of characters that differentiate this station from others.

makotech221 3y ago

Dunno about Go, but most c# solutions are around 2s and under https://hotforknowledge.com/2024/01/13/1brc-in-dotnet-among-fastest-on-linux-my-optimization-journey/#results https://hotforknowledge.com/2024/01/13/1brc-in-dotnet-among-...

junto 3y ago
Wow that’s pretty damn fast. C# has made some improvements in the past years or they have some other advantages?
- pjmlp 3y ago
  Yes to both. .NET team has been doubling down on performance improvements, people forget CLR also has features to support C like languages (hence Managed C++ and C++/CLI), and many of those capabilities are now surfaced into C# as well.
- Anon4Now 3y ago
  Stephen Toub wrote a book-length blog post about all the performance improvements made in .NET 8 [1]. Add the option to compile to native. [1] https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-8/ https://devblogs.microsoft.com/dotnet/performance-improvemen...
  - pjmlp 3y ago
    His blogs posts go back all the way to .NET 5, for those curious to some deep dive on performance improvements done on each release.
- to11mtm 3y ago
  Yeah. A few highlights, as can be seen in some of the blog posts mentioned by other replies: - 'Span<T>' to represent chunks of either managed OR unmanaged memory without using unsafe pointers throughout [0][1] - Not relevant to this task necessarily, but a lot of machinery has been added to allow reuse of objects for tasks like queuing thread pool work, or waiting for an asynchronous result. - Lots of intrinsics helpers for SIMD workloads, and increased usage of such intrinsics in internal parsers/etc. - Generally improving a lot of the internal IO to take advantage of other improvements in the runtime. - PGO (Performance Guided Optimization) on the JIT side, essentially helps with things like better devirt [2] and other improvements. - AOT compilation, if that's your thing, (I do believe the fastest C# 1BRC submissions use this) [0] - To be clear, unsafe can still be faster, however for most cases Span is fine and gives you a little more runtime safety. [1] - You can also grab a Span<T> of a primitive (i.e. int, char) within a method, so long as you don't blow up stack, this is very nice when you need a small buffer for parsing but don't want to thrash the GC or deal with volatile or locks on some sort of pool. [2] - Devirt historically was a problem in 'call heavy' .NET apps when Interfaces are used, before PGO there was more than one library I worked on where we intentionally used abstract base classes rather than interfaces due to the need to squeeze as much out as we could.

krallja 3y ago

There's a layer of pointer indirection when using slices in Go, you may be able to eke out some time by moving to arrays on the stack.

pram 3y ago

From looking at the final code it’s probably the performance of copy() as the biggest hurdle.

scotty79 3y ago

Instead of hash table I'd try sort of "eager" trie inside stack allocated memory. So I can find the slot for the stats of given station after parsing minimal number of characters that differentiate this station from others.