3 ms·
I implemented the 1brc in go a few weeks back [1] and this repo was inspiration to try to find a custom perfect hash function that put each station into its own
by infogulch 2y ago
I implemented the 1brc in go a few weeks back [1] and this repo was inspiration to try to find a custom perfect hash function that put each station into its own bucket with no collisions. Then I noticed the rule that you can't customize the hash function to the data before the program starts and dropped the idea.
I made a test harness to check random constants (start values, multiplication constants, shift/rotate amounts, etc) and print out the best constants found so far based on the number of colliding buckets and how many collisions. I think I got it down to 1 bucket with just two colliding values with a ~40% fill rate. Interestingly, I found that the best performing constants included similar values for the number of positions to shift regardless of the other constants, so I ended up hard coding them.
[1]: https://github.com/infogulch/1brc-go https://github.com/infogulch/1brc-go
- 0pteron 2y agojust FYI there are better techniques for finding "minimal perfect hash functions". This is an easy one to implement: https://cmph.sourceforge.net/chd.html https://cmph.sourceforge.net/chd.html