38 ms·

So you basically store S3 Buckets in Elastic Search and you're using Git for versioning a hierarchy of buckets, right? It's interesting that versioning now fin

So you basically store S3 Buckets in Elastic Search and you're using Git for versioning a hierarchy of buckets, right? It's interesting that versioning now finally seems to be getting some traction in mainstream database systems (even though they are not really optimal in these systems my opinion) and for instance also in your data store. You position this as a Dropbox or Google Drive replacement, right? :-) I'm asking all these questions, because I'm engineering a temporal, versioned Open Source storage system myself (since I studied at the University of Konstanz until 2012), possibly on a much more database oriented level -- currently for storing both XML and JSON data in a binary format. A resource in this storage system basically stores a huge tree of database pages whereas an UberPage is the main entry point (reminiscent of ZFSs UberPage, from which SirixDB borrows some ideas and puts these to the sub-file level), consisting of various more or less hash-array based subtrees as in ZFS. Thus, levels of indirect pages are added if more data needs to be stored. I've added some optimizations from in-memory hash-array based tries. Each revision is indexed. SirixDB stores per revision and per page deltas based on a copy-on-write log-structure. I've thought about storing each database page fragment in a S3 storage backend as another storage option and using Apache BookKeeper directly or Apache Pulsar for distributing an in-memory intent log (it doesn't need to be persisted before committing to the data files, as the UberPage just needs to be swapped atomically for consistency). For the interested reader: https://sirix.io https://sirix.io and https://github.com/sirixdb/sirix https://github.com/sirixdb/sirix

akarve 7y ago
Not quite ;) S3 is the primary data and metadata store, so that the rest of the stack is a pure function of S3 data (including Elastic). We don't use git at all yet. We use S3 object versioning and then capture the version, SHA-256, etag, etc. in a JSONL-based manifest https://open.quiltdata.com/b/quilt-example/tree/.quilt/packages/04de92de5f4c4141ee9c326a6719467293599b6478a6556e82d9f42a92440e6a https://open.quiltdata.com/b/quilt-example/tree/.quilt/packa.... Said JSONL manifest is simply a "locked list" of all the S3 objects in that package. The same manifests can be checked into git for fork/merge of data sets, but we're still exploring the right way to do that. I'll let Kevin answer the database fragments question.
- raghava 7y ago
  Neat. But would this not build dependency on s3s versioning and make it hard for getting this portable across other clouds?
  - akarve 7y ago
    Not quite. Abstraction layers like min.io support versioning. More importantly, Quilt manifests only require a "fully qualified physical key" that points to the data. In theory, the manifest can work with any URI: S3, local disk, etc.
kevinemoore 7y ago
Sirix sounds like a very interesting system! Is it similar in its internal structure to noms (https://github.com/attic-labs/noms https://github.com/attic-labs/noms) or Dolt (https://github.com/liquidata-inc/dolt https://github.com/liquidata-inc/dolt)? I think S3 is a good match for storing database pages as long as they are immutable. The Vectorized query processing model seems to fit this approach very well (e.g., http://oai.cwi.nl/oai/asset/14075/14075B.pdf http://oai.cwi.nl/oai/asset/14075/14075B.pdf) Anyone out there from Snowflake care to answer how Snowflake stores database pages in AWS? I haven't used BookKeeper or Pulsar myself so I can't comment on how well they might work for distributing an intent log.
- lichtenberger 7y ago
  I'm not really sure how Noms versions stuff. It seems they use a variant of a B-tree index. Or maybe only for secondary indexes? I think if you can use a simple monotonically increasing sequence number SirixDB has indexing advantages as for instance when storing XML and JSON documents or graph data. The cool thing also is that SirixDB not only copies changed database pages but it implements a sliding window algorithm for versioning the database pages itself along with well known backup versioning strategies. Furthermore user-specified, typed secondary index structures are also naturally versioned. One downside is that SirixDB doesn't support branching, even though it would be relatively easily possible to implement I guess, but I'm not convinced that it's needed. I don't want that anyone has to merge merge conflicts. I think automatic algorithms to do this are also not the right thing. But of course it's really interesting and I also thought about it :-) maybe someone has a really good use case? :-) BTW: Everything in SirixDB is immutable regarding updates of resources in databases. Of course you can revert to an old revision and change stuff, but the revisions in-between will still be accessible. Keep up the great work on Quilt :-)

akarve 7y ago

Not quite ;) S3 is the primary data and metadata store, so that the rest of the stack is a pure function of S3 data (including Elastic). We don't use git at all yet. We use S3 object versioning and then capture the version, SHA-256, etag, etc. in a JSONL-based manifest https://open.quiltdata.com/b/quilt-example/tree/.quilt/packages/04de92de5f4c4141ee9c326a6719467293599b6478a6556e82d9f42a92440e6a https://open.quiltdata.com/b/quilt-example/tree/.quilt/packa.... Said JSONL manifest is simply a "locked list" of all the S3 objects in that package. The same manifests can be checked into git for fork/merge of data sets, but we're still exploring the right way to do that. I'll let Kevin answer the database fragments question.

raghava 7y ago
Neat. But would this not build dependency on s3s versioning and make it hard for getting this portable across other clouds?
- akarve 7y ago
  Not quite. Abstraction layers like min.io support versioning. More importantly, Quilt manifests only require a "fully qualified physical key" that points to the data. In theory, the manifest can work with any URI: S3, local disk, etc.

kevinemoore 7y ago

Sirix sounds like a very interesting system! Is it similar in its internal structure to noms (https://github.com/attic-labs/noms https://github.com/attic-labs/noms) or Dolt (https://github.com/liquidata-inc/dolt https://github.com/liquidata-inc/dolt)? I think S3 is a good match for storing database pages as long as they are immutable. The Vectorized query processing model seems to fit this approach very well (e.g., http://oai.cwi.nl/oai/asset/14075/14075B.pdf http://oai.cwi.nl/oai/asset/14075/14075B.pdf) Anyone out there from Snowflake care to answer how Snowflake stores database pages in AWS? I haven't used BookKeeper or Pulsar myself so I can't comment on how well they might work for distributing an intent log.

lichtenberger 7y ago
I'm not really sure how Noms versions stuff. It seems they use a variant of a B-tree index. Or maybe only for secondary indexes? I think if you can use a simple monotonically increasing sequence number SirixDB has indexing advantages as for instance when storing XML and JSON documents or graph data. The cool thing also is that SirixDB not only copies changed database pages but it implements a sliding window algorithm for versioning the database pages itself along with well known backup versioning strategies. Furthermore user-specified, typed secondary index structures are also naturally versioned. One downside is that SirixDB doesn't support branching, even though it would be relatively easily possible to implement I guess, but I'm not convinced that it's needed. I don't want that anyone has to merge merge conflicts. I think automatic algorithms to do this are also not the right thing. But of course it's really interesting and I also thought about it :-) maybe someone has a really good use case? :-) BTW: Everything in SirixDB is immutable regarding updates of resources in databases. Of course you can revert to an old revision and change stuff, but the revisions in-between will still be accessible. Keep up the great work on Quilt :-)