3 ms·
I don't get this whole "never overwrite data" thing, including Datomic, for example. Isn't the disk space needed for these schemes enormous?
by fiatjaf 10y ago
I don't get this whole "never overwrite data" thing, including Datomic, for example.
Isn't the disk space needed for these schemes enormous?
- christophilus 10y agoI have the same question. Presumably, you could store fairly efficiently doing a git-like diff-based storage scheme or something. But I would be interested in hearing analysis of this.
- Salgat 10y agoExactly, events only represent changes in state. Furthermore, you aren't storing everything in the events; the business logic on services can add and build up models to much more than what the events hold (for example, an event could hold an id that the service uses to populate a bunch of properties into the actual model snapshot, something as simple as an order number would map to all kinds of order details).
- solidsnack9000 10y agoI have reservations, too: it's important to be able to remove data even though disk is cheap. * Removing very old data is a reasonable hedge for user privacy. * Sometimes confidential data makes its way into the data set and needs to be removed. * Old event data is often not useful but can impact performance or cost just the same. For example, one needs to allocate an EBS volume on AWS volume with a certain level of performance; but the cost of that is `IOPs * GBs`, not `IOPs * useful GBs`. * Replicating and backing up the dataset takes longer and longer as the application grows.
- yrashk 10y agoI agree, this is an importabt aspect. Our plan in PumpkinDB is to add key value association retirement, subject to defined retirement policies.