I know I mentioned this in another comment, but you really should check out Clickhouse. They have a table engine for exactly this purpose. You create a table w

I know I mentioned this in another comment, but you really should check out Clickhouse. They have a table engine for exactly this purpose. You create a table with the raw logs, then a materialized view (or another actual table) which declaratively does the rollup for you in real time. https://clickhouse.yandex/docs/en/table_engines/aggregatingmergetree/ https://clickhouse.yandex/docs/en/table_engines/aggregatingm... A full example: https://www.altinity.com/blog/2017/7/10/clickhouse-aggregatefunctions-and-aggregatestate https://www.altinity.com/blog/2017/7/10/clickhouse-aggregate...