4 ms·

This will be a tongue in cheek comment, but there's another thing Datomic isn't making you do either: GDPR compliance

by BenoitP 8y ago

davty 8y ago
Well, they do support extinctions?
- deleted 8y ago
  [deleted]
- blunte 8y ago
  That is certainly something an add-on or external tool could provide. Immutable is only immutable if you don't have write access to the lowest layer of storage.
  - augustl 8y ago
    It's difficult to implement this by hand, though. What you'll have in storage are encoded blobs of chunks of some kind of sorted set tree structure. So if you are to poke the data directly like that, I would assume it is much easier to just do an offline backup and a restore, and filter the data in the restore process somehow, which I believe is possible.
- emidln 8y ago
  Yes. They call this excision[0] and it leaves an Audi table hole in the timeline which lets you attach metadata like why there is missing data while still removing the data. [0] https://docs.datomic.com/on-prem/excision.html https://docs.datomic.com/on-prem/excision.html
  - Ryan1516 8y ago
    Excision is not support in Datomic Cloud
augustl 8y ago
OP here :) True, in that there's no solution out of the box. I kind of want to make a "look at all the things I have to do", and GDPR compliance is one of those things where there are probably many ways to do it wrong with Datomic, and a few ways of doing it right. One thing you can do, is to have a separate database for each person/user. It is trivial to join across multiple databases in Datomic, and you can even do crazy things like joining between a database and an excel spreadsheet. And deleting an entire database is easy. So there's that. There's also excision, but that's a super expensive operation that you shouldn't be doing as part of your day to day routines, according to the Datomic team. I'd like to know more here. For example, is it OK to excise data once a week? Maybe it's GDPR compliant if deletion requests are batched like that. There's also crypto shredding (encrypt values with separate key for each user, throw away the key on deletion request), but I'm not sure how GDPR compliant that is, since it leaves a lot of metadata behind. And you obviously can't encrypt values that you want to query on with the query engine.
- danielh 8y ago
  IANAL, but I‘m pretty sure it is acceptable to delete data within 30 days of receiving a request to do so.
  - augustl 8y ago
    Forgot to add. The built-in excision API is similar to crypto shredding, in that all the metadata is retained, it's just the values themselves (in Datomic's EAVT structure) that are removed. So a lot of metadata is still retained.
evanweaver 8y ago
For what it's worth, FaunaDB is also temporal and supports retention periods, TTL rows, and locality control, specifically for GDPR. Document-relational model instead of Datalog.
blunte 8y ago
Gpdr compliance is as much as not choosing to retrieve data after a certain point as it is not possessing that data. It's a bit of an existential point, but if your logic allows the disclusion of data before a certain point of time, is that not effectively the same as not possessing it?

This will be a tongue in cheek comment, but there's another thing Datomic isn't making you do either: GDPR compliance

davty 8y ago

Well, they do support extinctions?

deleted 8y ago
[deleted]
blunte 8y ago
That is certainly something an add-on or external tool could provide. Immutable is only immutable if you don't have write access to the lowest layer of storage.
- augustl 8y ago
  It's difficult to implement this by hand, though. What you'll have in storage are encoded blobs of chunks of some kind of sorted set tree structure. So if you are to poke the data directly like that, I would assume it is much easier to just do an offline backup and a restore, and filter the data in the restore process somehow, which I believe is possible.
emidln 8y ago
Yes. They call this excision[0] and it leaves an Audi table hole in the timeline which lets you attach metadata like why there is missing data while still removing the data. [0] https://docs.datomic.com/on-prem/excision.html https://docs.datomic.com/on-prem/excision.html
- Ryan1516 8y ago
  Excision is not support in Datomic Cloud

augustl 8y ago

OP here :) True, in that there's no solution out of the box. I kind of want to make a "look at all the things I have to do", and GDPR compliance is one of those things where there are probably many ways to do it wrong with Datomic, and a few ways of doing it right. One thing you can do, is to have a separate database for each person/user. It is trivial to join across multiple databases in Datomic, and you can even do crazy things like joining between a database and an excel spreadsheet. And deleting an entire database is easy. So there's that. There's also excision, but that's a super expensive operation that you shouldn't be doing as part of your day to day routines, according to the Datomic team. I'd like to know more here. For example, is it OK to excise data once a week? Maybe it's GDPR compliant if deletion requests are batched like that. There's also crypto shredding (encrypt values with separate key for each user, throw away the key on deletion request), but I'm not sure how GDPR compliant that is, since it leaves a lot of metadata behind. And you obviously can't encrypt values that you want to query on with the query engine.

danielh 8y ago
IANAL, but I‘m pretty sure it is acceptable to delete data within 30 days of receiving a request to do so.
- augustl 8y ago
  Forgot to add. The built-in excision API is similar to crypto shredding, in that all the metadata is retained, it's just the values themselves (in Datomic's EAVT structure) that are removed. So a lot of metadata is still retained.

evanweaver 8y ago

For what it's worth, FaunaDB is also temporal and supports retention periods, TTL rows, and locality control, specifically for GDPR. Document-relational model instead of Datalog.

blunte 8y ago

Gpdr compliance is as much as not choosing to retrieve data after a certain point as it is not possessing that data. It's a bit of an existential point, but if your logic allows the disclusion of data before a certain point of time, is that not effectively the same as not possessing it?