3 ms·
This was a very similar conversation that the team at my last job had. Our data were customer-facing so it was a slightly different problem, but the question of
by ebiester 3y ago
This was a very similar conversation that the team at my last job had. Our data were customer-facing so it was a slightly different problem, but the question of "our application cares about what the present state is, but our analytics cares about trends and point-in-time" is a more universal problem.
We used Debezium and Change Data Capture + Outbox patterns in some cases, and solely outbox patterns in others. (See https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/ https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/ ) - You then can use the change events to do the analysis. That solves your "log all actions" problem. It's still a lot of work to put together the analysis (a tool like DBT helps) but it is a solved problem with enough labor at that point and doesn't disrupt the actual application.
If I was starting from scratch on an application today that I thought might have analytics needs in the future, I would get this early on even if I didn't need it yet.
- playing_colours 3y agoDid you roll out your own implementation of Outbox pattern? It is an important pattern to address potential data inconsistencies, but I did not see yet a good scalable implementation of it.
- ebiester 3y agoWe rolled our own. I think a lot of it can be helped with Debezium (and we leaned on it, if I remember correctly.)