3 ms·

I submitted a kinda related link (counting 128M distinct objects) a few weeks ago, which might be of interest: Realtime Metrics for 128 Million Users with R

• Groxx 15y ago
  The technique is very different, though - the getspool article makes a 128 million bit bitmask and flips a bit for each user. This article refers to probabilistic techniques for counting number-of-unique objects, and uses massively less memory to do so. The bitmask technique gives 128 million in 16MB RAM and no error, where the HyperLogLog technique gives 1,000 million in 0.0015MB RAM and 2% margin of error.