4 ms·
A way to deal with it that I've seen to work with Kafka is to make your messages not last very long (say about a week) and make them explicitly idempotent. So a
by seer 7y ago
A way to deal with it that I've seen to work with Kafka is to make your messages not last very long (say about a week) and make them explicitly idempotent. So an "old" message being run would not affect the app negatively.
And then you make every producer of messages able to reproduce all the messages it knows about.
So if you have a new service and need historical data, you ask all of your dependencies to resend the data, and existing services should not be affected.
Schema evolution is natural - old messages with old schemas don't last very long, and you can slowly migrate services to new schemas as needed.
If you structure your events that each one holds all state of an entity, you could get removal of data easily for free, as any new message would overwrite old state.
Though the whole thing has its own problems though - especially when you go into large amounts of messages/data. Keeping copies of those around if you make a lot of changes can be quite expensive.
So the whole event sourcing thing looks to me as it needs a decade or two to mature so that tools are built and best practices established. It looks to me as basically a way to keep data for a lot of separate services in a fluid eventually consistent way.
Wonder if Clojure's Datomic is already there. Haven't used it myself but have read promising things about it.
- randolorian 7y agoBest practices around this have already been established. Most if not all event stores - which Kafka is not - have a concept called 'position.' You save the position atomically along with whatever you did with the message. Then if you crash, you simply ask for all messages starting from that position. If you have a new service (or a new projection), your position is 0 so you get everything.
- seer 7y agoThat is indeed the case, and this is what Kafka calls unlimited retention topics. This however hampers schema evolution significantly as you're not allowed to make backwards incompatible changes. Or rather if you allow those changes, every service would need to be able to handle every schema throughout time. If you make the retention to only several days this would mean you can evolve your schema with even breaking changes. Consumers would need to support only schemas that are "active right now". If the retention period is long enough that it is safe to assume all the consumers have acted on all the old messages you can delete those messages. You would need a mechanism to replay them though, if that data is ever needed again, but it would just be in the latest schema.
- randolorian 7y agoI feel like people are running into these problems because they want to pretend that a message broker is an event store. I could try to shovel a star schema into MongoDB too, but why would I want to? Keeping data only in the latest schema is dangerous. We have no idea what data the business will find useful years down the line. By only having whatever is in the latest schema, you may have thrown valuable data away.
- lukev 7y agoDatomic is interesting in the context of event sourcing, because it can be thought of as a highly generalized event store system itself, with each database transaction being an event.