If you're interested, I built a stupid simple deduplication algorithm[1]. It ironically uses the avalanche effect of another hash function to implement random

If you're interested, I built a stupid simple deduplication algorithm[1]. It ironically uses the avalanche effect of another hash function to implement random projection onto a locality preserving bit vector. It sounds like real computer science bingo but it's actually surprisingly simple and it works much better than it has any business doing. [1] https://github.com/MarginaliaSearch/MarginaliaSearch/blob/master/code/libraries/easy-lsh/src/main/java/nu/marginalia/lsh/EasyLSH.java https://github.com/MarginaliaSearch/MarginaliaSearch/blob/ma...