4 ms·

I'm not a Kafka user. However, I am a long-time keen implementer of event sourcing and log-derived data stores. Many of these involve per-user or per-individual

by dpwm 9y ago

I'm not a Kafka user. However, I am a long-time keen implementer of event sourcing and log-derived data stores. Many of these involve per-user or per-individual sqlite databases. I cannot understand why Kafka is causing people problems. I am surprised that this seems to be being accepted as a problem with event sourcing in general. The most obvious solution I came up with with GDPR right to be forgotten was to have per-user logs. User gone? Delete their log. Then you just need to handle how you share the projected data from the individual event logs. Messages are unlikely to be spread over multiple log events and can be just sourced directly. It's clear who owns that data: it's the user who wrote the message as they can at any time delete the only copy. Aggregation is more difficult but can be done if you structure the things you are aggregating such that you only need a few lookups per user. Again you can cache references to certain events that can be efficiently looked up using a (user_id, event_id) pair. There's nothing to stop you storing the latest projection in a separate table of the per-user database file. Is there some technical reason why so many Kafka users seem to be having such difficulty with this?

Heliosmaster 9y ago
Your idea is right, but it really depends on the event-store implementation. For example, Kafka has a limited amount of topics (10K if I remember well), so unless you have a very limited audience, I don't think having one topic per user is the solution over there.
- dpwm 9y ago
  Right. This makes sense as to why it is a problem for Kafka users. I must admit I was in a bit of a hurry when I looked into Kafka to see if it would be a better fit, but it seemed to be more about the scaling than I cared for.
noir_lord 9y ago
What (if any) is your favourite book on event sourcing? I've seen tech meetup lectures (one particulary fascinating one was a guy who worked for the NHS) but I'd like to understand it better than I do.
- dpwm 9y ago
  To me the biggest thing that made event sourcing click was when I was working on an internal app that needed full auditing. With event sourcing the audit log is the authoritative data source. Learning that there was a name for this and that others had worked it out better than I had was very handy. I didn't find a need for a book because I found reading Martin Fowler's article[0] on it to give me a good enough understanding. I've found that it's a less well-defined pattern than some implementations make it seem. There are many trade-offs to be made. At the one extreme you can treat events as physical events that can be easily understood. Let's say we have a user that wants to add a phone number. Well, we could have an AddPhoneNumber event. But this leads to a RemovePhoneNumber event as well. But you can also take the other extreme and say that each event is a set of collected information -- perhaps a different element of the root of a json object -- and that you can just diff to see what changed and only need to look at the latest such event. I've found there's a middle ground that doesn't require too much thinking: related data goes together if it's not going to change very often. So I will generally put all contact information in together as an UpdateContacts event. This way we don't need to implement all the list operations for phone numbers and can skip looking for such events after the latest has been found. You also still have something simple enough to work out what changed and that is unlikely to fill your storage. If you are implementing event sourcing, I would like to point out one thing that I didn't see written anywhere: set a reasonable upper limit on per-user storage. Because you will find somebody who, even not maliciously, just keeps changing things back and forth and it's better to stop them adding events than to stop all users by running out of disk space. [0] https://martinfowler.com/eaaDev/EventSourcing.html https://martinfowler.com/eaaDev/EventSourcing.html
  - noir_lord 9y ago
    Thank you for the thoughtful response. I was hoping for a good book of the "here be dragons" kind. Often books on an architecture gloss over the rough bits and then you find them yourself when you already committed to it. I've become incredibly conservative when it comes to application architecture the older I get.
    - 9y ago
nbevans 9y ago
Indeed - we designed ours in a similar way as you describe and whilst GDPR causes a small bit of work for us it is not like we face a total design flaw type issue that it appears people are experiencing with Kafka. We are looking at adding a Nuke command + Nuked event that will serve to destroy all data in the aggregate root snapshot and leave behind (effectively) a tombstone snapshot. The command handler will then purge all prior events leaving behind just a Nuked event. This means the event stream doesn't just disappear entirely, it can still be synchronised out to dependent systems, and indeed those systems will learn of the nukage just like any other event.
alsor847 9y ago
By default Kafka has no mechanism to delete records from the event store, apart from a TTL. You can enable topic compaction which can be used to delete records by overwriting them, but compacted topics just don't make sense for a lot of Kafka use cases. So the only option left is the one suggested in this article: encrypt everything with individual keys and delete the key (rather than the data) to comply with GPDR
beepbeepbeep1 9y ago
Kafaka you get topics and partitions. Topics is a grouping of data, normally having a common schema. Partitions allow parallelism spreading the data across multiple brokers. Data on a topic can be partitioned by a key so you can partition by user id resulting in all user events going to the same partition however there’s no functionality to delete a partition by key. If each user has their own topic that you can delete you would end up with a lot of topics and Kafka isn’t great with that. You also have the issue if event sourcing or doing DDD your events are based around domains/aggregates with clear context bounds. The domain is very likely not a user but may involve a user. Its a tricky modeling issue. Event sourcing is about immutable events. You cannot change an event in the past as its already happened, accounts don’t use rubbers. The ability to go back and change past events or delete them breaks the laws and guarantees of an immutable event system. It’s a difficult problem. I like event sourcing due to the properties it gives but it may not be suitable for some things due to GDPR. One thing I have heard talked about is encrypting personal identifiable information in an event with a user specific key. When the user requests deletion you delete the key. The events don’t change they are still there however anything which identifies a user is unreadable.

by dpwm 9y ago

Heliosmaster 9y ago

Your idea is right, but it really depends on the event-store implementation. For example, Kafka has a limited amount of topics (10K if I remember well), so unless you have a very limited audience, I don't think having one topic per user is the solution over there.

dpwm 9y ago
Right. This makes sense as to why it is a problem for Kafka users. I must admit I was in a bit of a hurry when I looked into Kafka to see if it would be a better fit, but it seemed to be more about the scaling than I cared for.

noir_lord 9y ago

What (if any) is your favourite book on event sourcing? I've seen tech meetup lectures (one particulary fascinating one was a guy who worked for the NHS) but I'd like to understand it better than I do.

dpwm 9y ago
To me the biggest thing that made event sourcing click was when I was working on an internal app that needed full auditing. With event sourcing the audit log is the authoritative data source. Learning that there was a name for this and that others had worked it out better than I had was very handy. I didn't find a need for a book because I found reading Martin Fowler's article[0] on it to give me a good enough understanding. I've found that it's a less well-defined pattern than some implementations make it seem. There are many trade-offs to be made. At the one extreme you can treat events as physical events that can be easily understood. Let's say we have a user that wants to add a phone number. Well, we could have an AddPhoneNumber event. But this leads to a RemovePhoneNumber event as well. But you can also take the other extreme and say that each event is a set of collected information -- perhaps a different element of the root of a json object -- and that you can just diff to see what changed and only need to look at the latest such event. I've found there's a middle ground that doesn't require too much thinking: related data goes together if it's not going to change very often. So I will generally put all contact information in together as an UpdateContacts event. This way we don't need to implement all the list operations for phone numbers and can skip looking for such events after the latest has been found. You also still have something simple enough to work out what changed and that is unlikely to fill your storage. If you are implementing event sourcing, I would like to point out one thing that I didn't see written anywhere: set a reasonable upper limit on per-user storage. Because you will find somebody who, even not maliciously, just keeps changing things back and forth and it's better to stop them adding events than to stop all users by running out of disk space. [0] https://martinfowler.com/eaaDev/EventSourcing.html https://martinfowler.com/eaaDev/EventSourcing.html
- noir_lord 9y ago
  Thank you for the thoughtful response. I was hoping for a good book of the "here be dragons" kind. Often books on an architecture gloss over the rough bits and then you find them yourself when you already committed to it. I've become incredibly conservative when it comes to application architecture the older I get.
  - 9y ago

nbevans 9y ago

Indeed - we designed ours in a similar way as you describe and whilst GDPR causes a small bit of work for us it is not like we face a total design flaw type issue that it appears people are experiencing with Kafka. We are looking at adding a Nuke command + Nuked event that will serve to destroy all data in the aggregate root snapshot and leave behind (effectively) a tombstone snapshot. The command handler will then purge all prior events leaving behind just a Nuked event. This means the event stream doesn't just disappear entirely, it can still be synchronised out to dependent systems, and indeed those systems will learn of the nukage just like any other event.

alsor847 9y ago

By default Kafka has no mechanism to delete records from the event store, apart from a TTL. You can enable topic compaction which can be used to delete records by overwriting them, but compacted topics just don't make sense for a lot of Kafka use cases. So the only option left is the one suggested in this article: encrypt everything with individual keys and delete the key (rather than the data) to comply with GPDR

beepbeepbeep1 9y ago

Kafaka you get topics and partitions. Topics is a grouping of data, normally having a common schema. Partitions allow parallelism spreading the data across multiple brokers. Data on a topic can be partitioned by a key so you can partition by user id resulting in all user events going to the same partition however there’s no functionality to delete a partition by key. If each user has their own topic that you can delete you would end up with a lot of topics and Kafka isn’t great with that. You also have the issue if event sourcing or doing DDD your events are based around domains/aggregates with clear context bounds. The domain is very likely not a user but may involve a user. Its a tricky modeling issue. Event sourcing is about immutable events. You cannot change an event in the past as its already happened, accounts don’t use rubbers. The ability to go back and change past events or delete them breaks the laws and guarantees of an immutable event system. It’s a difficult problem. I like event sourcing due to the properties it gives but it may not be suitable for some things due to GDPR. One thing I have heard talked about is encrypting personal identifiable information in an event with a user specific key. When the user requests deletion you delete the key. The events don’t change they are still there however anything which identifies a user is unreadable.