3 ms·
AWS Athena is pretty good at tackling this problem, or PrestoDB running on EMR. As long as your S3 data is reasonably partitioned, and you don't have millions
by djhworld 8y ago
AWS Athena is pretty good at tackling this problem, or PrestoDB running on EMR.
As long as your S3 data is reasonably partitioned, and you don't have millions of small files, it does a reasonable job -even on count(distinct)
It even support approx_distinct for hyperloglog estimation too