3 ms·
I think streaming-first systems like Flink are best positioned to answer your need. You might need to abstract the query or subscription infrastructure into yo
by BenoitP 5y ago
I think streaming-first systems like Flink are best positioned to answer your need.
You might need to abstract the query or subscription infrastructure into your data model though:
Subscriptions:
INSERT INTO user_orders_total_feed(user_id, data)
SELECT
uo.user_id,
SUM(o.price) total
FROM user_order uo
JOIN order o ON uo.user_id = o.user_id
JOIN user_live_subscription uls ON uo.user_id = uls.user_id
GROUP BY uo.user_id
WHERE uls.subscription_topic = 'orders_total'
And for queries you would join on a temporal snapshot using 'FOR SYSTEM TIME AS OF' [1]
You could have your data fully stored in Kafka or Pulsar, Kappa-architecture-style ; or you could have it in regular rdbms but with Change Data Capture on top [2] (basically shipping your WAL to Kafka/Pulsar)
The good thing about Kafka and Flink is that they scale. Your regular db might not sustain the query rate if every query needs a write to a table, but Kafka/Flink will.
[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/joins.html#processing-time-temporal-join https://ci.apache.org/projects/flink/flink-docs-stable/dev/t...
[2] https://debezium.io/ https://debezium.io/