6 ms·
The last section on Operational Flexibility and the inability to change the event history raises a very good point. Like most of the issues, the solution requi
by lscharen 8y ago
The last section on Operational Flexibility and the inability to change the event history raises a very good point.
Like most of the issues, the solution requires experience to know when you are at the Goldilocks point (Just Right). This specific issue has a lot in common with managing database migrations in django or any other migration system.
The ideal situation is to create migrations that can always be rolled back, but sometimes this is not possible to do operationally. For example, a schema change that restricts a field from NVARCHAR to INTEGER can only be generically rolled back if all of the unconvertable data is persisted. This can be mitigated by structuring the database to avoid these dead-ends, and that really is only gained by hard-won experience.
The problem with undoing operations via new events is the same thing -- unless you have foreknowledge of this kind of problem, it is very easy to accidentally create events that perform un-undoable actions. A very simple example of a problematic event would be something that modifies a foreign-key relationship -- let's say it's a digital asset in a game and you want the ability to transfer ownership of the item from one player to another.
The simple solution is a event like SetAssetOwner(asset_pk, player_pk). This would set the item's player foreign key field to the player's primary key. Easy. However, you have lost knowledge of where the asset came from and cannot undo this operation. A better solution would be to make an event SwapAssetOwner(asset_pk, owner_pk, recipient_pk). Yes, the owner_pk is technically redundant, but it provides a check against someone trying to steal an item with a maliciously crafted SetAssetOwner event that performs no checking. Even better, this operation can be reverted by sending the same event with the owner and recipient arguments reversed. Since these properties are part of the event message, they will be persisted in the event history and all of the information to undo the event is self-contained.
- spitfire 8y agoBasically implement double entry accounting. I think ideally you want something like Rich Hickey speaks about when he speaks of Datomic. An append only database. You can see what the previous values for that row were, along with schema changes.
- fusiongyro 8y agoI worry about GDPR with respect to Kafka and other append-only databases.
- einrealist 8y agoYou can encrypt events and throw away the keys, if data should be made inaccessible. Of course, it adds complexity. But its already being done.
- jandrewrogers 8y agoThis does not work for many data models, for both technical and economic (e.g. increasing costs by multiple orders of magnitude) reasons. Many real systems would require hundreds of millions of active encryption keys, encrypting data that is smaller than the encryption block size. Every database architecture that exists today is designed with the deep assumption that scalable fine-grained deletion will never be required, largely because we don't have good computer science for how to do it. As experienced database operations people know, if you are required to do this kind of delete then the sane way is to completely rebuild your storage with the data to be deleted filtered out, if you have ample excess capacity -- it is often much faster than editing the existing storage. For some large-scale systems, there is no plausible solution. This is an interesting computer science challenge -- a database kernel designed for efficient deletion -- that I've thought a lot about over the last year dealing with GDPR compliance. It definitely isn't a thing that exists today and encryption doesn't help.
- solipsism 8y agoEvery database architecture that exists today is designed with the deep assumption that scalable fine-grained deletion will never be required, largely because we don't have good computer science for how to do it Can you explain this? Are you talking about something different from deleting individual rows?
- 8y ago
- Yokohiii 8y agoIdk why people are saying that ES data is always immutable. They can be by default sure, but if a facility is useful to change the history, why not?
- hesdeadjim 8y agoIf you need to change history you can just create new events that accomplish your mutation -- and even mark them as a type 'change history' or some such obvious identifier so when you inspect the event stream you know exactly what you are looking at.
- ht85 8y agoTo avoid having to implement the whole logic for any event you'd want to revert, you can have a "cancel" event that points to another event. You'd read the cancel events first and simply skip the events they point to.
- jacques_chester 8y ago> If you need to change history you can just create new events that accomplish your mutation -- and even mark them as a type 'change history' or some such obvious identifier so when you inspect the event stream you know exactly what you are looking at. "Right to be forgotten" isn't the same as "right to be redacted".
- mianos 8y agoI agree, this is 100% normal in accounting, as the earlier thread pointed out. If there is an error you add journal entries to the end to make the adjustment. It is funny that this sort of thing was invented in accounting in 1494.
- wst_ 8y agoIn many cases they are legal proof that something happened. If you can mess with history, proof can't be used in court.