4 ms·
Yahoo has a full library of production Sketches; that is, fast approximate algorithms for computing stats on large datasets. https://datasketches.github.io/ ht
by wenc 9y ago
Yahoo has a full library of production Sketches; that is, fast approximate algorithms for computing stats on large datasets.
https://datasketches.github.io/ https://datasketches.github.io/
Hyperloglog for fast COUNT DISTINCTs is included in this library, as well as quantile sketches, reservoir sampling sketches, theta sketches, etc.
https://datasketches.github.io/docs/TheChallenge.html https://datasketches.github.io/docs/TheChallenge.html
- Radim 9y agoAre there any benchmarks for this Yahoo lib? To trade performance/accuracy numbers, rather than impressions. There's no shortage of various streamed counting implementations; the sweet spot of Bounter [0], the Python lib from the original blog post, is its super simple use (drop-in replacement for Python's built-in Counter), small memory footprint & high performance (optimized C underneath). [0] https://github.com/RaRe-Technologies/bounter https://github.com/RaRe-Technologies/bounter