3 ms·
Great work! I haven't messed around with the count-min augmentation for HLLs, but I've had a lot of success in practice. I've experimented with using HLL sket
by stochastic_monk 9y ago
Great work!
I haven't messed around with the count-min augmentation for HLLs, but I've had a lot of success in practice.
I've experimented with using HLL sketches for set operations for genome comparisons. The great thing about these operations is that the cost of operations scales with the size of the sketch, not the size of the dataset being sketched. Add SIMD acceleration, and you have an extremely fast, compact, accurate way to perform approximate set operations.
Unfortunately, the naive intersection operation (a sketch consisting of an element-wise 'min' of the counts in both sketches) does not perform well in cardinality estimation.
The recent Ertl paper [0] and associated code [1] puts a lot of effort into more accurate estimation and set operations. In my experiments, I've found that his modified estimation formula is always more accurate than the standard method and remains accurate to much higher cardinalities for a given sketch size.
My SIMD-accelerated, threadsafe HyperLogLog implementation in C++ (with python bindings) using this improved estimation method is available at [2]. Using this structure, I was able to perform over a billion full-genome Jaccard index calculations overnight using `bonsai dist` from [3].
[0]: https://arxiv.org/pdf/1702.01284.pdf https://arxiv.org/pdf/1702.01284.pdf
[1]: https://github.com/oertl/hyperloglog-sketch-estimation-paper https://github.com/oertl/hyperloglog-sketch-estimation-paper
[2]: https://github.com/dnbaker/hll https://github.com/dnbaker/hll
[3]: https://github.com/dnbaker/bonsai https://github.com/dnbaker/bonsai
- xstartup 9y agoHow do you find the relevant papers? Is there any tool for this task?
- dman 9y agoGoogle?
- stochastic_monk 9y agoTo be honest, a combination of arxiv automailings, colleagues, and github. A lot of people I work with are interested in sketch data structures.