4 ms·

Hi there -- We are currently ingesting 150M/day -- though it's not evenly distributed -- probably peaks around 3,000qps Agreed that it's not too much yet. I th

by LogicX 9y ago

Hi there -- We are currently ingesting 150M/day -- though it's not evenly distributed -- probably peaks around 3,000qps Agreed that it's not too much yet. I think the element which kills us is needing to do rollup tables, and then query against both the raw data and rollup data for our customer analytics dashboard. The data is a combination of date/time, strings, and ints. 22 fields.

ryanworl 9y ago
I know I mentioned this in another comment, but you really should check out Clickhouse. They have a table engine for exactly this purpose. You create a table with the raw logs, then a materialized view (or another actual table) which declaratively does the rollup for you in real time. https://clickhouse.yandex/docs/en/table_engines/aggregatingmergetree/ https://clickhouse.yandex/docs/en/table_engines/aggregatingm... A full example: https://www.altinity.com/blog/2017/7/10/clickhouse-aggregatefunctions-and-aggregatestate https://www.altinity.com/blog/2017/7/10/clickhouse-aggregate...