4 ms·
We encounter a similar problem at my current job (mixing systems that we want to keep stateful with systems that we want to make “real-time”/stream-based). I t
by gen220 7y ago
We encounter a similar problem at my current job (mixing systems that we want to keep stateful with systems that we want to make “real-time”/stream-based).
I think you’ve covered most of the problems you’ll encounter. One thing that sticks out to me is downtime: how will your order subscriber handle a product publisher that’s down or otherwise delayed? Then, the events will be potentially out of order, is that a problem for you?
On another note, we follow the same bounded context principles, but we implemented it with Kafka+confluent, since that infrastructure and those libraries were already available. Teams make their data accessible via a mix of “raw” Kafka topics and very refined gRPC services. Your subscriber is implemented as a cron job that reads from N stream and “reduces” them to 1 stream.
FWIW, we also store a transaction log in each of our databases, so we can generate a stream of object states relatively easily later on. This has helped a lot with converting old tables into streams, and vice versa.
The only thing that’s a persistent issue is schema changes. My only recommendation there is to never make them... In all seriousness, keep your data models small, and whenever you want to experiment with a schema change, add the new data as a FK’d table with its own transaction log, rather than a schema mutation to your core table. It’s never worth the headache if you take data integrity seriously.