5 ms·

I looked through the docs and these are the main points of "bi-temporal" and "unbundled": > Record the true history of your business whilst also preserving an

by memexy 6y ago

I looked through the docs and these are the main points of "bi-temporal" and "unbundled": > Record the true history of your business whilst also preserving an immutable transaction record - this is the essence of bitemporality. Unlock the ability to efficiently query through time, as a developer and as an application user. Crux enables you to create retroactive corrections, simplify historical data migrations and build an integrated view of out-of-order event data. > Apache Kafka for the primary storage of transactions and documents as semi-immutable logs. > RocksDB or LMDB to host indexes for rich query support.

Scarbutt 6y ago
Sounds good except for "is a document db".
- memexy 6y ago
  What's wrong with document DBs?
  - adamkl 6y ago
    Document DBs are ok if you only ever want to use the data in the same way that you stored it. There are definitely use cases where that makes sense, but you lose a lot of query power when compared to a relational database, which makes it less adaptable to changing requirements. That being said, I believe (it’s been a while since I looked at Crux), it actually maintains a separate set of Entity-Value-Attribute indices generated from the contents of the immutable series of documents stored in Kafka/RocksDB. This should give you similar level of query power when compared to something like Datomic. Disclaimer, I haven’t had any actual hands-on experience with either Crux or Datomic, so I could be wrong.
    - refset 6y ago
      That's a pretty accurate summary. Documents are very straightforward to reason about at a transactional level, particularly when handling unpredictable changes to data models. Once documents are ingested into the EAV-like indexes you can query everything much the same as a typical graph database, because all the ID references within the documents translate into edges within a schemaless graph of entities. Crux's native query language is a flavour of Datalog but we also have a SQL layer in the works using Apache Calcite, for a more traditional relational feel.
  - tannhaeuser 6y ago
    Not GP but my guess is that "document DB" is really a term that can mean all things to all people, and is even used as a trademark (by AWS as name for their Mongo-API-compatible product). As used in this context, "JSON store" would be more appropriate maybe. I know that in academic DB literature, "document DB" refers to any storage that leaves the original serialization intact (if it has a serialized form to begin with). But outside of that context, if people were to google "document database", they probably commonly expect to find a product that can store, retrieve, fulltext-search, version-control, manage the access permissions and transform pipelines of real documents, those being either arbitrary binary document formats, or more specifically markup (XML, SGML, HTML) documents. I'd also suggest vendors come up with more precise nomenclature in their own best interest, because the way it's now is that laypeople will be directed towards the products with the most ad spent rather than what they're looking for in the majority of cases.
    - refset 6y ago
      These are definitely valid points, thanks. I would like to see Crux evolve into something more comprehensive like MarkLogic eventually, partly to not disappoint anyone with those common expectations of what a "document database" ought to have, but also because it feels like the right direction to move in if we want to see an offline-first world succeed in the future...and pick-up where Lotus Notes stopped innovating! If we had native JSON support already I suspect we would have opted for "JSON Store", but we've only been able to support edn since we launched and "edn Store" isn't too helpful. An official Lucene index is very likely going to be added for full-text search at some point. I know of two users in the community who have already integrated Lucene themselves, which is actually fairly straightforward as long as you don't need to do searches in the past or can cope with returning results from across all of time.
      - tannhaeuser 6y ago
        MarkLogic today advertise with data integration and NoSQL though rather than being a "document DB", which I guess is even worse as a moniker.
- NicoJuicy 6y ago
  Perhaps you don't know enough about the subject ;) I too had that opinion once, untill I started real DDD and event sourcing. An opinion can change quickly ;)
  - cbushko 6y ago
    Could you expand on this a bit or point to articles that talk about it? It sounds interesting
    - refset 6y ago
      I can only guess what was meant, but I suspect the main argument is that projection code in an event sourced system is much more nimble when it's not tied down to an underlying schema. Documents make focusing on language level types much easier. That said, we've not built Crux with domain-level event sourcing in mind in particular. There is certainly overlap in the bitemporal model Crux has and what I gather some people are doing with retroactive events. This blog post on Datomic and event sourcing is interesting and relevant for Crux also (Crux has been heavily inspired by Datomic): https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...
    - NicoJuicy 6y ago
      I would advise you to look up domain events and follow the rabbit hole from there ;) Perhaps "ddd quickly" is a good 66 page summary.
      - refset 6y ago
        Are there any tools you recommend for DDD? Have you seen much justification for bitemporal or retroactive events, as in https://dddeurope.com/2018/speakers/thomas-pierrain/ https://dddeurope.com/2018/speakers/thomas-pierrain/ ?
refset 6y ago
It's worth noting that Crux supports a healthy range of options for transaction log and document storage beyond Kafka, including a variety of JDBC backends (SQLite, Postgres etc.) Kafka is usually more than most people want or need. There was a blog post recently about a new Firebase-like Clojure web framework which uses Crux + Digital Ocean's managed Postgres service: https://findka.com/blog/migrating-to-biff/ https://findka.com/blog/migrating-to-biff/ Disclosure: working on Crux :)
- yellowapple 6y ago
  > including a variety of JDBC backends (SQLite, Postgres etc.) How straightforward would it be to add a new JDBC-based node implementation? I'm noticing that SQL Server is apparently missing, which is a bit of a shame given how many off-the-shelf products use it (and could therefore probably integrate with Crux if Crux did so, too).
  - refset 6y ago
    It's usually no more than a day or two's work for us to get a new dialect figured out and to wire up CI tests with containers etc. It wouldn't be infeasible for someone new to Clojure to manage it about the same time also. The hard part is the SQL, not the Clojure. Fortunately there actually is SQL Server support already: https://github.com/juxt/crux/blob/master/crux-jdbc/src/crux/jdbc/mssql.clj https://github.com/juxt/crux/blob/master/crux-jdbc/src/crux/... ...and I'll make sure the full range of backends is made clearer somewhere. Good feedback!

by memexy 6y ago

Scarbutt 6y ago

Sounds good except for "is a document db".

memexy 6y ago
What's wrong with document DBs?
- adamkl 6y ago
  Document DBs are ok if you only ever want to use the data in the same way that you stored it. There are definitely use cases where that makes sense, but you lose a lot of query power when compared to a relational database, which makes it less adaptable to changing requirements. That being said, I believe (it’s been a while since I looked at Crux), it actually maintains a separate set of Entity-Value-Attribute indices generated from the contents of the immutable series of documents stored in Kafka/RocksDB. This should give you similar level of query power when compared to something like Datomic. Disclaimer, I haven’t had any actual hands-on experience with either Crux or Datomic, so I could be wrong.
  - refset 6y ago
    That's a pretty accurate summary. Documents are very straightforward to reason about at a transactional level, particularly when handling unpredictable changes to data models. Once documents are ingested into the EAV-like indexes you can query everything much the same as a typical graph database, because all the ID references within the documents translate into edges within a schemaless graph of entities. Crux's native query language is a flavour of Datalog but we also have a SQL layer in the works using Apache Calcite, for a more traditional relational feel.
- tannhaeuser 6y ago
  Not GP but my guess is that "document DB" is really a term that can mean all things to all people, and is even used as a trademark (by AWS as name for their Mongo-API-compatible product). As used in this context, "JSON store" would be more appropriate maybe. I know that in academic DB literature, "document DB" refers to any storage that leaves the original serialization intact (if it has a serialized form to begin with). But outside of that context, if people were to google "document database", they probably commonly expect to find a product that can store, retrieve, fulltext-search, version-control, manage the access permissions and transform pipelines of real documents, those being either arbitrary binary document formats, or more specifically markup (XML, SGML, HTML) documents. I'd also suggest vendors come up with more precise nomenclature in their own best interest, because the way it's now is that laypeople will be directed towards the products with the most ad spent rather than what they're looking for in the majority of cases.
  - refset 6y ago
    These are definitely valid points, thanks. I would like to see Crux evolve into something more comprehensive like MarkLogic eventually, partly to not disappoint anyone with those common expectations of what a "document database" ought to have, but also because it feels like the right direction to move in if we want to see an offline-first world succeed in the future...and pick-up where Lotus Notes stopped innovating! If we had native JSON support already I suspect we would have opted for "JSON Store", but we've only been able to support edn since we launched and "edn Store" isn't too helpful. An official Lucene index is very likely going to be added for full-text search at some point. I know of two users in the community who have already integrated Lucene themselves, which is actually fairly straightforward as long as you don't need to do searches in the past or can cope with returning results from across all of time.
    - tannhaeuser 6y ago
      MarkLogic today advertise with data integration and NoSQL though rather than being a "document DB", which I guess is even worse as a moniker.
NicoJuicy 6y ago
Perhaps you don't know enough about the subject ;) I too had that opinion once, untill I started real DDD and event sourcing. An opinion can change quickly ;)
- cbushko 6y ago
  Could you expand on this a bit or point to articles that talk about it? It sounds interesting
  - refset 6y ago
    I can only guess what was meant, but I suspect the main argument is that projection code in an event sourced system is much more nimble when it's not tied down to an underlying schema. Documents make focusing on language level types much easier. That said, we've not built Crux with domain-level event sourcing in mind in particular. There is certainly overlap in the bitemporal model Crux has and what I gather some people are doing with retroactive events. This blog post on Datomic and event sourcing is interesting and relevant for Crux also (Crux has been heavily inspired by Datomic): https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...
  - NicoJuicy 6y ago
    I would advise you to look up domain events and follow the rabbit hole from there ;) Perhaps "ddd quickly" is a good 66 page summary.
    - refset 6y ago
      Are there any tools you recommend for DDD? Have you seen much justification for bitemporal or retroactive events, as in https://dddeurope.com/2018/speakers/thomas-pierrain/ https://dddeurope.com/2018/speakers/thomas-pierrain/ ?

refset 6y ago

It's worth noting that Crux supports a healthy range of options for transaction log and document storage beyond Kafka, including a variety of JDBC backends (SQLite, Postgres etc.) Kafka is usually more than most people want or need. There was a blog post recently about a new Firebase-like Clojure web framework which uses Crux + Digital Ocean's managed Postgres service: https://findka.com/blog/migrating-to-biff/ https://findka.com/blog/migrating-to-biff/ Disclosure: working on Crux :)

yellowapple 6y ago
> including a variety of JDBC backends (SQLite, Postgres etc.) How straightforward would it be to add a new JDBC-based node implementation? I'm noticing that SQL Server is apparently missing, which is a bit of a shame given how many off-the-shelf products use it (and could therefore probably integrate with Crux if Crux did so, too).
- refset 6y ago
  It's usually no more than a day or two's work for us to get a new dialect figured out and to wire up CI tests with containers etc. It wouldn't be infeasible for someone new to Clojure to manage it about the same time also. The hard part is the SQL, not the Clojure. Fortunately there actually is SQL Server support already: https://github.com/juxt/crux/blob/master/crux-jdbc/src/crux/jdbc/mssql.clj https://github.com/juxt/crux/blob/master/crux-jdbc/src/crux/... ...and I'll make sure the full range of backends is made clearer somewhere. Good feedback!