4 ms·
Since you asked about long term storage and occasional queries, have you considered offloading to the cloud? (Disclosure: I work for Google) With BigQuery you
by fhoffa 10y ago
Since you asked about long term storage and occasional queries, have you considered offloading to the cloud?
(Disclosure: I work for Google)
With BigQuery you can:
- Store as much relational data as you want ($20->$10 per terabyte per month).
- Query it whenever, and get charged only by query (no servers needed)
- To stay on topic: HyperLogLog:
https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#hyperloglog-functions https://cloud.google.com/bigquery/docs/reference/standard-sq...
Note that BigQuery exposes partial HLL functions like HLL_COUNT.INIT, HLL_COUNT.MERGE, HLL_COUNT.MERGE_PARTIAL, and HLL_COUNT.EXTRACT.
"These functions operate on sketches that compress an arbitrary set into a fixed-memory representation. The sketches can be merged to produce a new sketch that represents the union, before a final estimate is extracted from the sketch."
- lima 10y agoFor now we're trying to do it "on premise" since we're a large-ish service provider with operations and hardware teams. As long as the operational burden isn't too high, we do it ourselves to gain experiences with different technologies. However, BigQuery is really tempting since it'd probably solve all of the issues we're currently having. Will definitely do some benchmarking. The pricing is hard to beat .
- vgt 10y agoBigQuery is ALWAYS the right choice :) (a very biased Googler perspective) In all seriousness, BigQuery is amazing - hit me up if you have any questions!