4 ms·

GDPR and Kappa/event sourcing/message queue based/you name it architecture goes together nicely as you get audit logs of everything and it should be quite doabl

GDPR and Kappa/event sourcing/message queue based/you name it architecture goes together nicely as you get audit logs of everything and it should be quite doable to propagate "delete this person's data" events around the place. It's a huge hassle compared to what many companies are doing with customer data now but I think it's for the best. Most things about GDPR go like "Does it feel a bit shady? It probably is. Don't do that." (depending on your moral compass of course) One thing is for sure: there's a lot of opportunities for consultants as all the big companies need help to resolve the mess of legacy systems storing customer data.

TobbenTM 9y ago
(assuming you meant to write Kafka) Being able to notify every internal service to delete a user's data is always nice, but in the case of event sourcing, the events are the data. Yet you can't delete Kafka events (not sure about other platforms). In my eyes, GDPR is the death of Kafka as an event sourcing store.
- elnygren 9y ago
  Kafka doesn't persist things forever. AFAIK it's totally OK if you have say 14d retention and then the data is deleted from Kafka too (it was deleted from everywhere else already because there was an event/request to do so)
  - TobbenTM 9y ago
    Heh, I think we're talking about different scenarios. In the case of event sourcing, we often set the retention period to 'forever', because the events in Kafka are our source of truth. Then we just build a materialising layer on top of Kafka, with the possibility to rehydrate based on _every_ event in the Kafka topics. In this case we would have to do some really weird compaction to delete singular events.
    - dpwm 9y ago
      It really depends on what you're doing. If you're doing something where the people you're storing data about don't need to interact with each other, you can actually store each user's events in a separate event log. This can be fed to more transient event queues which do not have an indefinite retention period where interaction is required. Not sure about how well-geared Kafka is to this scenario though.
    - dominotw 9y ago
      I think that scenario is adressed on this confulent blog post > Deleting a message from a compacted topic is as simple as writing a new message to the topic with the key you want to delete and a null value. When compaction runs the message will be deleted forever. Handling GDPR with Apache Kafka: How does a log forget? https://www.confluent.io/blog/handling-gdpr-log-forget/ https://www.confluent.io/blog/handling-gdpr-log-forget/
      - TobbenTM 9y ago
        Oh, wow, must have missed that post, thanks!
- bonesss 9y ago
  Scenario specific: with Kafkas log compaction you could use message keys and republish a stripped message to the old key, preserving the history and non-personal information, but keeping the queue and message series intact...
espadrine 9y ago
I didn't realize it until reading this post, but certain very popular technologies break GDPR in a deep way. Bitcoin, for instance, contains a wealth of personal information, which by design are both public, persisted forever, and immutable. Are blockchain products all going to need a full rewrite or a complicated hard fork? What about the Wayback Machine? Will they need to have an endpoint that every company will need to call for every “right to be forgotten” request worldwide?
- kalefranz 9y ago
  I think the dependency order here is reversed. GDPR breaks certain very popular technologies in a deep way.
- pbhjpbhj 9y ago
  What personal information does bitcoin contain?
  - espadrine 9y ago
    All payment orders and credit transfers to and from all accounts. For any Bitcoin address you find on the Web.
    - sb8244 9y ago
      Is Bitcoin identifiable? Or is an association to a Bitcoin transaction in a payment system identifiable? Which would mean the personal info could be removed in the payment system
    - pbhjpbhj 9y ago
      How is that personal, how is it connected to a person?
      - espadrine 9y ago
        On the off-chance that you are not asking this in bad faith… All speculators go through a KYC with their exchange, which identifies them very precisely. All other users paste it publicly, saying “I own this account! Send me money.” And even those users often have to convert it back to fiat, which requires an exchange, which makes them go through a KYC.

TobbenTM 9y ago

(assuming you meant to write Kafka) Being able to notify every internal service to delete a user's data is always nice, but in the case of event sourcing, the events are the data. Yet you can't delete Kafka events (not sure about other platforms). In my eyes, GDPR is the death of Kafka as an event sourcing store.

elnygren 9y ago
Kafka doesn't persist things forever. AFAIK it's totally OK if you have say 14d retention and then the data is deleted from Kafka too (it was deleted from everywhere else already because there was an event/request to do so)
- TobbenTM 9y ago
  Heh, I think we're talking about different scenarios. In the case of event sourcing, we often set the retention period to 'forever', because the events in Kafka are our source of truth. Then we just build a materialising layer on top of Kafka, with the possibility to rehydrate based on _every_ event in the Kafka topics. In this case we would have to do some really weird compaction to delete singular events.
  - dpwm 9y ago
    It really depends on what you're doing. If you're doing something where the people you're storing data about don't need to interact with each other, you can actually store each user's events in a separate event log. This can be fed to more transient event queues which do not have an indefinite retention period where interaction is required. Not sure about how well-geared Kafka is to this scenario though.
  - dominotw 9y ago
    I think that scenario is adressed on this confulent blog post > Deleting a message from a compacted topic is as simple as writing a new message to the topic with the key you want to delete and a null value. When compaction runs the message will be deleted forever. Handling GDPR with Apache Kafka: How does a log forget? https://www.confluent.io/blog/handling-gdpr-log-forget/ https://www.confluent.io/blog/handling-gdpr-log-forget/
    - TobbenTM 9y ago
      Oh, wow, must have missed that post, thanks!
bonesss 9y ago
Scenario specific: with Kafkas log compaction you could use message keys and republish a stripped message to the old key, preserving the history and non-personal information, but keeping the queue and message series intact...

espadrine 9y ago

I didn't realize it until reading this post, but certain very popular technologies break GDPR in a deep way. Bitcoin, for instance, contains a wealth of personal information, which by design are both public, persisted forever, and immutable. Are blockchain products all going to need a full rewrite or a complicated hard fork? What about the Wayback Machine? Will they need to have an endpoint that every company will need to call for every “right to be forgotten” request worldwide?

kalefranz 9y ago
I think the dependency order here is reversed. GDPR breaks certain very popular technologies in a deep way.
pbhjpbhj 9y ago
What personal information does bitcoin contain?
- espadrine 9y ago
  All payment orders and credit transfers to and from all accounts. For any Bitcoin address you find on the Web.
  - sb8244 9y ago
    Is Bitcoin identifiable? Or is an association to a Bitcoin transaction in a payment system identifiable? Which would mean the personal info could be removed in the payment system
  - pbhjpbhj 9y ago
    How is that personal, how is it connected to a person?
    - espadrine 9y ago
      On the off-chance that you are not asking this in bad faith… All speculators go through a KYC with their exchange, which identifies them very precisely. All other users paste it publicly, saying “I own this account! Send me money.” And even those users often have to convert it back to fiat, which requires an exchange, which makes them go through a KYC.