4 ms·

A huge practical challenge when implementing bitemporality is how to deal with schema migrations efficiently whilst supporting arbitrary time-travel queries. Wh

by refset 7y ago

A huge practical challenge when implementing bitemporality is how to deal with schema migrations efficiently whilst supporting arbitrary time-travel queries. When we were building Crux, which is referenced at the end of the post, we took the view that this is a hard problem to solve in a generic way, hence why we have opted for a schemaless document-oriented design. I really hope to see other databases make it easier to use bitemporality in the near future, but I suspect that any DBMS which mandates a schema is fighting an up-hill battle. Disclosure: working on Crux at JUXT :)

paulcarey 7y ago
You could consider ‘partitioning’ your DB when you migrate your schema so each DB instance only deals with a single bitemporal slice eg after 5 migrations you have 5 distinct database instances. This approach avoids bloat assuming the vast majority of queries would be served by the most recent instance, while not precluding serving verbatim responses from earlier instances.
- lichtenberger 7y ago
  I wonder why so few people seem to think about persistent data structures (persistent in the functional sense, that is immutability -- copy-on-write). Basically we implemented a form of hash array based tries for https://sirix.io https://sirix.io ... or it's just like how ZFS stores objects adding levels of indirect blocks/pages on demand with bitsets storing which indexes are really set to avoid having arrays with a lot of null references. For sure you have to copy the path to the root, just like snapshots work in ZFS or Btrfs, but I think that data is neglegable when you basically also do versioning on a per record-level/per revision level. At least thus we can restore each revision in the same time, you do not have to have a "history table", which is inefficient to query in contrast to the most recent revision table.
  - lichtenberger 7y ago
    But it's schema-less ;-)

by refset 7y ago

paulcarey 7y ago

You could consider ‘partitioning’ your DB when you migrate your schema so each DB instance only deals with a single bitemporal slice eg after 5 migrations you have 5 distinct database instances. This approach avoids bloat assuming the vast majority of queries would be served by the most recent instance, while not precluding serving verbatim responses from earlier instances.

lichtenberger 7y ago
I wonder why so few people seem to think about persistent data structures (persistent in the functional sense, that is immutability -- copy-on-write). Basically we implemented a form of hash array based tries for https://sirix.io https://sirix.io ... or it's just like how ZFS stores objects adding levels of indirect blocks/pages on demand with bitsets storing which indexes are really set to avoid having arrays with a lot of null references. For sure you have to copy the path to the root, just like snapshots work in ZFS or Btrfs, but I think that data is neglegable when you basically also do versioning on a per record-level/per revision level. At least thus we can restore each revision in the same time, you do not have to have a "history table", which is inefficient to query in contrast to the most recent revision table.
- lichtenberger 7y ago
  But it's schema-less ;-)