4 ms·
I've done it both ways. Look into Data Sketches also if you want to see applications. The pros: -- Samples are small and fast most of the time. -- can be use
by kwillets 2y ago
I've done it both ways. Look into Data Sketches also if you want to see applications.
The pros:
-- Samples are small and fast most of the time.
-- can be used opportunistically, eg in queries against the full dataset.
-- can run more complex queries that can't be pre-aggregated (but not always accurately).
The cons:
-- requires planning about what to sample and what types of queries you're answering. Sudden requirements changes are difficult.
-- data skew makes uniform sampling a bad choice.
-- requires ETL pipelines to do the sampling as new data comes in. That includes re-running large backfills if data or sampling changes.
-- requires explaining error to users
-- Data sketches can be particularly inflexible; they're usually good at one metric but can't adapt to new ones. Queries also have to be mapped into set operations.
These problems can be mitigated with proper management tools; I have built frameworks for this type of application before -- fixed dashboards with slow-changing requirements are relatively easy to handle.