4 ms·

It would be interesting to hear about the use-cases that prompted the creation of Sirix. For example, we were inspired to build Crux[0], another bitemporal docu

by refset 7y ago

It would be interesting to hear about the use-cases that prompted the creation of Sirix. For example, we were inspired to build Crux[0], another bitemporal document database (although we opted for Datalog rather that XQuery), following our experiences of integrating timestamped data from multiple upstream systems, whilst coping with delays and ad-hoc corrections, and also maintaining efficient time-travel auditability. [0] https://juxt.pro/crux https://juxt.pro/crux

lichtenberger 7y ago
Some of the use cases can be found here: https://sirix.io/documentation.html https://sirix.io/documentation.html The main distinctive features are our main document store index, features inherited by ZFS (checksums in parent pointers, the main index, a trie, compression and hopefully soon encryption of page-fragments, always consistent on-disk, log-structured without the need of a WAL...), versioned user-defined indexes, highly concurrent data structures (every transaction has access to one snapshot and we only allow one read/write transaction, parallelization has to be done by the client code) and record-level versioning (also a novel sliding window algorithm -- whereas a slightly other implementation is patented by the founder Marc Kramis). I might implement pointer swizzling for the upcoming 1.0.0 release, should speedup Sirix considerably :-) https://sirix.io/features.html https://sirix.io/features.html I think XQuery is great for querying JSON data. That said in the future I'd also implement something based on Spark to distribute queries...
- refset 7y ago
  Cool, thanks for the summary! The user-defined indexes sound particularly intriguing. The documentation page alludes to payroll, audit and decision support applications -- have you implemented one or more of those already? I have been compiling a list of known uses for bitemporality in the Crux docs which you are more than welcome to borrow from: https://juxt.pro/crux/docs/bitemp.html#_known_uses https://juxt.pro/crux/docs/bitemp.html#_known_uses It is great to see all this new enthusiasm for temporal databases now that they are finally viable, decades after all the major research happened :)
  - lichtenberger 7y ago
    Will have to sleep now... but yes, the ideas and the first prototype emerged already in I think 2006 from Marc Kramis (back then it internally was named Idefix, then Treetank... but I think Treetank is pretty strange ;-))
juskrey 7y ago
How is Crux different from Datomic?
- slifin 7y ago
  These are the differences I think I know about so far: - You can't lie to Datomic, it's immutable and has some really interesting performance characteristics because of that i.e. TTL = infinity, the downside from a legal pov is no information is really gone only redacted from now, I think the law should be updated so that redacted is enough (if not already, I'm not a lawyer) Crux can delete things not sure about performance ramifications - Datomic stores information as datoms which are single pieces of information which are easy to re-arrange at run time to be any shape of tree you like, Crux is a document store which is a small tree already, not sure what abilities there are to re-arrange the tree at query time - storing business time in Datomic is inefficient because it has to be added to the transaction which I don't think is added to indexes, this is the primary use case of crux though - cardinality is defined up front in Datomic which I suspect catches data integrity errors, I haven't seen anything in crux to support this - crux doesn't support either the pull or entity syntax ( I don't know which one or why that's important, I'm a bit out of my depth now)
  - lichtenberger 7y ago
    Wow, great summary, thanks from me, too :)

by refset 7y ago

lichtenberger 7y ago

Some of the use cases can be found here: https://sirix.io/documentation.html https://sirix.io/documentation.html The main distinctive features are our main document store index, features inherited by ZFS (checksums in parent pointers, the main index, a trie, compression and hopefully soon encryption of page-fragments, always consistent on-disk, log-structured without the need of a WAL...), versioned user-defined indexes, highly concurrent data structures (every transaction has access to one snapshot and we only allow one read/write transaction, parallelization has to be done by the client code) and record-level versioning (also a novel sliding window algorithm -- whereas a slightly other implementation is patented by the founder Marc Kramis). I might implement pointer swizzling for the upcoming 1.0.0 release, should speedup Sirix considerably :-) https://sirix.io/features.html https://sirix.io/features.html I think XQuery is great for querying JSON data. That said in the future I'd also implement something based on Spark to distribute queries...

refset 7y ago
Cool, thanks for the summary! The user-defined indexes sound particularly intriguing. The documentation page alludes to payroll, audit and decision support applications -- have you implemented one or more of those already? I have been compiling a list of known uses for bitemporality in the Crux docs which you are more than welcome to borrow from: https://juxt.pro/crux/docs/bitemp.html#_known_uses https://juxt.pro/crux/docs/bitemp.html#_known_uses It is great to see all this new enthusiasm for temporal databases now that they are finally viable, decades after all the major research happened :)
- lichtenberger 7y ago
  Will have to sleep now... but yes, the ideas and the first prototype emerged already in I think 2006 from Marc Kramis (back then it internally was named Idefix, then Treetank... but I think Treetank is pretty strange ;-))

juskrey 7y ago

How is Crux different from Datomic?

slifin 7y ago
These are the differences I think I know about so far: - You can't lie to Datomic, it's immutable and has some really interesting performance characteristics because of that i.e. TTL = infinity, the downside from a legal pov is no information is really gone only redacted from now, I think the law should be updated so that redacted is enough (if not already, I'm not a lawyer) Crux can delete things not sure about performance ramifications - Datomic stores information as datoms which are single pieces of information which are easy to re-arrange at run time to be any shape of tree you like, Crux is a document store which is a small tree already, not sure what abilities there are to re-arrange the tree at query time - storing business time in Datomic is inefficient because it has to be added to the transaction which I don't think is added to indexes, this is the primary use case of crux though - cardinality is defined up front in Datomic which I suspect catches data integrity errors, I haven't seen anything in crux to support this - crux doesn't support either the pull or entity syntax ( I don't know which one or why that's important, I'm a bit out of my depth now)
- lichtenberger 7y ago
  Wow, great summary, thanks from me, too :)