3 ms·

What's wrong with document DBs?

by memexy 6y ago

adamkl 6y ago
Document DBs are ok if you only ever want to use the data in the same way that you stored it. There are definitely use cases where that makes sense, but you lose a lot of query power when compared to a relational database, which makes it less adaptable to changing requirements. That being said, I believe (it’s been a while since I looked at Crux), it actually maintains a separate set of Entity-Value-Attribute indices generated from the contents of the immutable series of documents stored in Kafka/RocksDB. This should give you similar level of query power when compared to something like Datomic. Disclaimer, I haven’t had any actual hands-on experience with either Crux or Datomic, so I could be wrong.
- refset 6y ago
  That's a pretty accurate summary. Documents are very straightforward to reason about at a transactional level, particularly when handling unpredictable changes to data models. Once documents are ingested into the EAV-like indexes you can query everything much the same as a typical graph database, because all the ID references within the documents translate into edges within a schemaless graph of entities. Crux's native query language is a flavour of Datalog but we also have a SQL layer in the works using Apache Calcite, for a more traditional relational feel.
tannhaeuser 6y ago
Not GP but my guess is that "document DB" is really a term that can mean all things to all people, and is even used as a trademark (by AWS as name for their Mongo-API-compatible product). As used in this context, "JSON store" would be more appropriate maybe. I know that in academic DB literature, "document DB" refers to any storage that leaves the original serialization intact (if it has a serialized form to begin with). But outside of that context, if people were to google "document database", they probably commonly expect to find a product that can store, retrieve, fulltext-search, version-control, manage the access permissions and transform pipelines of real documents, those being either arbitrary binary document formats, or more specifically markup (XML, SGML, HTML) documents. I'd also suggest vendors come up with more precise nomenclature in their own best interest, because the way it's now is that laypeople will be directed towards the products with the most ad spent rather than what they're looking for in the majority of cases.
- refset 6y ago
  These are definitely valid points, thanks. I would like to see Crux evolve into something more comprehensive like MarkLogic eventually, partly to not disappoint anyone with those common expectations of what a "document database" ought to have, but also because it feels like the right direction to move in if we want to see an offline-first world succeed in the future...and pick-up where Lotus Notes stopped innovating! If we had native JSON support already I suspect we would have opted for "JSON Store", but we've only been able to support edn since we launched and "edn Store" isn't too helpful. An official Lucene index is very likely going to be added for full-text search at some point. I know of two users in the community who have already integrated Lucene themselves, which is actually fairly straightforward as long as you don't need to do searches in the past or can cope with returning results from across all of time.
  - tannhaeuser 6y ago
    MarkLogic today advertise with data integration and NoSQL though rather than being a "document DB", which I guess is even worse as a moniker.
    - refset 6y ago
      Indeed, it's an impressively eclectic product, considering how niche it seems to still be. Incidentally they have some fairly nice materials about their own bitemporal query support and it's use across industries, e.g.: https://www.marklogic.com/blog/bitemporal/ https://www.marklogic.com/blog/bitemporal/ https://www.marklogic.com/resources/mifid-ii-its-about-data-not-reporting/ https://www.marklogic.com/resources/mifid-ii-its-about-data-...

What's wrong with document DBs?

by memexy 6y ago

What's wrong with document DBs?

Document DBs are ok if you only ever want to use the data in the same way that you stored it. There are definitely use cases where that makes sense, but you lose a lot of query power when compared to a relational database, which makes it less adaptable to changing requirements. That being said, I believe (it’s been a while since I looked at Crux), it actually maintains a separate set of Entity-Value-Attribute indices generated from the contents of the immutable series of documents stored in Kafka/RocksDB. This should give you similar level of query power when compared to something like Datomic. Disclaimer, I haven’t had any actual hands-on experience with either Crux or Datomic, so I could be wrong.

refset 6y ago
That's a pretty accurate summary. Documents are very straightforward to reason about at a transactional level, particularly when handling unpredictable changes to data models. Once documents are ingested into the EAV-like indexes you can query everything much the same as a typical graph database, because all the ID references within the documents translate into edges within a schemaless graph of entities. Crux's native query language is a flavour of Datalog but we also have a SQL layer in the works using Apache Calcite, for a more traditional relational feel.

tannhaeuser 6y ago

Not GP but my guess is that "document DB" is really a term that can mean all things to all people, and is even used as a trademark (by AWS as name for their Mongo-API-compatible product). As used in this context, "JSON store" would be more appropriate maybe. I know that in academic DB literature, "document DB" refers to any storage that leaves the original serialization intact (if it has a serialized form to begin with). But outside of that context, if people were to google "document database", they probably commonly expect to find a product that can store, retrieve, fulltext-search, version-control, manage the access permissions and transform pipelines of real documents, those being either arbitrary binary document formats, or more specifically markup (XML, SGML, HTML) documents. I'd also suggest vendors come up with more precise nomenclature in their own best interest, because the way it's now is that laypeople will be directed towards the products with the most ad spent rather than what they're looking for in the majority of cases.

refset 6y ago
These are definitely valid points, thanks. I would like to see Crux evolve into something more comprehensive like MarkLogic eventually, partly to not disappoint anyone with those common expectations of what a "document database" ought to have, but also because it feels like the right direction to move in if we want to see an offline-first world succeed in the future...and pick-up where Lotus Notes stopped innovating! If we had native JSON support already I suspect we would have opted for "JSON Store", but we've only been able to support edn since we launched and "edn Store" isn't too helpful. An official Lucene index is very likely going to be added for full-text search at some point. I know of two users in the community who have already integrated Lucene themselves, which is actually fairly straightforward as long as you don't need to do searches in the past or can cope with returning results from across all of time.
- tannhaeuser 6y ago
  MarkLogic today advertise with data integration and NoSQL though rather than being a "document DB", which I guess is even worse as a moniker.
  - refset 6y ago
    Indeed, it's an impressively eclectic product, considering how niche it seems to still be. Incidentally they have some fairly nice materials about their own bitemporal query support and it's use across industries, e.g.: https://www.marklogic.com/blog/bitemporal/ https://www.marklogic.com/blog/bitemporal/ https://www.marklogic.com/resources/mifid-ii-its-about-data-not-reporting/ https://www.marklogic.com/resources/mifid-ii-its-about-data-...