I think approximate solutions would be fine as long as they're an underestimate. As long as the estimate is close enough, the company shouldn't be bothered by t

I think approximate solutions would be fine as long as they're an underestimate. As long as the estimate is close enough, the company shouldn't be bothered by the lost revenue. As long as it never overestimates, the customer can't complain. HyperLogLog is too inaccurate to be practical. But, if these files are smaller than a few billion lines, a Bloom Filter could get 0.0001% accuracy while still fitting in main memory on a single node.