Author suggests a data store without explicitly explaining why; reason being the benchmarks focused on time, not space at scale, but the context assumes needing

Author suggests a data store without explicitly explaining why; reason being the benchmarks focused on time, not space at scale, but the context assumes needing to scalable. At scale, HyperLogLog appears to be a good choice for managing space demands. This paper covers analysis of HyperLogLog (HLL): http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf TLDR: Calculating the exact cardinality of a multiset requires an amount of memory proportional to the cardinality, which is impractical for very large data sets. Probabilistic cardinality estimators, such as the HyperLogLog algorithm, use significantly less memory than this, at the cost of obtaining only an approximation of the cardinality. The HyperLogLog algorithm is able to estimate cardinalities of > 10^9 with a typical accuracy (standard error) of 2%, using 1.5 kB of memory. Source: https://en.wikipedia.org/wiki/HyperLogLog https://en.wikipedia.org/wiki/HyperLogLog