3 ms·

Great work and very interesting ideas. I'm working on a versioned database system[1] which offers similar features and benefits: - storage engine written f

Great work and very interesting ideas. I'm working on a versioned database system[1] which offers similar features and benefits: - storage engine written from scratch - completely isolated read-only transactions and one read/write transaction concurrently with a single lock to guard the writer. Readers will never be blocked by the single read/write transaction and execute without any latches/locks. - variable sized pages - lightweight buffer management with a "kind of" pointer swizzling - dropping the need for a write ahead log due to atomic switching of an UberPage - rolling merkle hash tree of all nodes built during updates optionally - ID-based diff-algorithm to determine differences between revisions taking the (secure) hashes optionally into account - non-blocking REST-API, which also takes the hashes into account to throw an error if a subtree has been modified in the meantime concurrently during updates - versioning through a huge persistent and durable, variable sized page tree using copy-on-write - storing delta page-fragments using a patented sliding snapshot algorithm - using a special trie, which is especially good for storing records sith numerical dense, monotonically increasing 64 Bit integer IDs. We make heavy use of bit shifting to calculate the path to fetch a record - time or modification counter based auto commit - versioned, user-defined secondary index structures - a versioned path summary - indexing every revision, such that a timestamp is only stored once in a RevisionRootPage. The resources stored in SirixDB are based on a huge, persistent (functional) and durable tree - sophisticated time travel queries As I'm spending a lot of my spare time on the project and would love to spend even more time, give it a try :-) Any help is more than welcome. Kind regards Johannes [1] https://sirix.io https://sirix.io and https://github.com/sirixdb/sirix https://github.com/sirixdb/sirix

erichocean 7y ago
> - completely isolated read-only transactions and one read/write transaction concurrently with a single lock to guard the writer. Readers will never be blocked by the single read/write transaction and execute without any latches/locks. > - variable sized pages > - lightweight buffer management with a "kind of" pointer swizzling > - dropping the need for a write ahead log due to atomic switching of an UberPage LMDB made those same design choices and is extremely fast/robust.
- lichtenberger 7y ago
  In my particular case it was also a design decision made back in 2006 or 2007 already. It's designed for fast random reads from the ground up due to the versioning focus (reading page-fragments from different revisions, as it just stores fragments of record-pages). I'll change the algorithm slightly to fetch the fragments in parallel (should be fast on modern hardware, that is even SSDs and in the future for instance with byte-addressable non-volatile memory).

erichocean 7y ago

> - completely isolated read-only transactions and one read/write transaction concurrently with a single lock to guard the writer. Readers will never be blocked by the single read/write transaction and execute without any latches/locks. > - variable sized pages > - lightweight buffer management with a "kind of" pointer swizzling > - dropping the need for a write ahead log due to atomic switching of an UberPage LMDB made those same design choices and is extremely fast/robust.

lichtenberger 7y ago
In my particular case it was also a design decision made back in 2006 or 2007 already. It's designed for fast random reads from the ground up due to the versioning focus (reading page-fragments from different revisions, as it just stores fragments of record-pages). I'll change the algorithm slightly to fetch the fragments in parallel (should be fast on modern hardware, that is even SSDs and in the future for instance with byte-addressable non-volatile memory).