4 ms·

Fun tutorial! When does it make sense to use Datomic over something like MySQL or PostgreSQL?

abraxas 4y ago
Mostly when you need a ledger like functionality to your data. Datomic preserves the history of all the changes by design so it is very easy to reconstruct the history of changes to your data. If this describes the majority of your problem space then Datomic is a good fit. Otherwise I would stick to a popular RDBMS like Postgres to leverage the community behind it. You will have a much easier time interacting with a well understood open source engine.
- emccue 4y ago
  I think this is true - you will have an easier time not blazing new trails in general - but the need of ledger-like functionality isn't the distinguishing factor. Preserving the history of all changes isn't just for reconstructing that history later, it's a central component in maintaining ACID while allowing for distributed readers. Datomic has a singular transactor - node that can accept writes. So all transactions happen in sequence and consistency is maintained within the data model. So your database goes DB (0 transactions) -> DB' (1 transaction committed) -> DB'' (2 transactions committed) With normal databases, when DB'' is created, DB' is lost. You change stuff in place. This means that anyone wanting to observe a consistent view of the database needs to coordinate with writes. This is why SQL dbs are single giant machines. For Cassandra, Dynamo, Mongo, etc you still change stuff in place, but you just accept that things will be potentially inconsistent ("eventually consistent") for your readers. Once you drop that part of ACID you can scale your readers and your writers arbitrarily. With Datomic DB, DB', and DB'' are still maintained. If you want to read the database "now" you can get a handle to the state that your reader thinks is the most recent. This might not be behind the absolute newest state that the transactor knows about, but it is still a consistent view of the database. Its because of this you can scale your read nodes independently of the system. Because if you are reading DB' you know the value of that won't change you can load chunks of the indexes into the memory for caching and even embed the querying into the application making queries. The proprietary-ness and JVM-ness of Datomic are its biggest weaknesses. I choose to believe those weaknesses can be addressed.
  - huahaiy 4y ago
    Every database that implements MVCC does this, that means most ACID databases. So "reading a consistent view" is not unique for Datomic.
  - Nihilartikel 4y ago
    I'm waiting to have a need to use it, but XTDB is a nice fully open bitemporal 'lite' Datomic that I have been keeping an eye on.
  - runekaagaard 4y ago
    Ah yes, makes me remember the amazing talk by Rich Hickey called "The database as a Value" [1]. I felt like my brain was reorganized after watching it. The simplicity of Datomic is very appealing, it's just a timestamped series of EAV entries, and some fancy indexing. Avoiding the latency (the database over there issue) sounds like a big win. Most of the simplicity is made possible by only having one single synchronous writer which is a major drawback you have to be able to live with. Some write-heavy applications would struggle. Related to the timetravel feature I've bookmarked a blog post by Valentin Waeselynck where he talks about some of the misconceptions and drawbacks. [2]. My company have a not super-great setup with triggers on MySQL, and have longed for MariaDBs SQL 2011 compliant Temporal Tables which looks very interesting [3]. I guess an advantage with Datomic is that you don't have to create an entire table history row when all you want is to update a single field. I was interested enough in Datomic that I started building a Python bridge but never finished it [4]. An added bonus was that I learned about the Transit serialization format which was ALSO super inspiring [5]. [1] https://www.youtube.com/watch?v=D6nYfttnVco [2] https://vvvvalvalval.github.io/posts/2017-07-08-Datomic-this-is-not-the-history-youre-looking-for.html [3] https://mariadb.com/resources/blog/temporal-tables-part-1/ [4] https://github.com/runekaagaard/brygge/blob/master/src/jolle/example.py [5] https://cognitect.com/blog/2014/7/22/transit
- john567 4y ago
  This is true but it's a false premise. You have history but it's clunky to work with. You won't be able to "source" and recreate the past without some work and unexpected things will happen when you "source" the past with the most recent code. It useful but the history aspect is easily misunderstood.
  - emccue 4y ago
    In this respect XTDB has a more directly useful model
  - blain_the_train 4y ago
    Can you explain what "source" means?
    - john567 4y ago
      Oh, yeah, think of event sourcing. You have a series of events leading up (aggregate) into some state. This can be done in different ways but with Datomic you can request a point in time database. This database doesn't exist so it needs to be "sourced" from history. This means running through everything that happened up to some point to create a snapshot of the database state (or query).
emccue 4y ago
As the world stands today, it makes sense to use Datomic if you are writing code on the JVM and are willing to pay for the database and associated support. In a hypothetical future world where the data model and query approaches have more widespread open source adoption it is competitive for a lot of the same use cases.
casion 4y ago
In addition to what was said, Datomic Cloud automates a lot of the annoying and difficult to suss deployment for application dev, not just the database. You can deploy large scalable apps in minutes and easily deploy new code in minutes with no down time (or architecting a deployment system yourself).
alecco 4y ago
Datomic is elegant and it has some interesting properties like temporal. This is a huge win for many cases where business logic needs them (e.g. finance). I think it would make more sense for clients also running JVM. I don't have much experience with it but it looks very good. -- a SQL engine dev
john567 4y ago
What I've find most useful is the immutable nature of the database. That within a reactive framework for UX is great combination. You get really cool things to happen with little effort. Also, because the database is a value. You can easily do mock data and isolated transactions locally to experiment and test. This makes some things easier. Another cool thing is how you do cross database queries by simply passing multiple databases to a function. It will pull in the subset of the dataset needed to compute the query result. It's that easy to do. It's not good at dealing with high volume transactions (a lot of writes) it's not too bad but it's not built for that primarily.
- amelius 4y ago
  > What I've find most useful is the immutable nature of the database. Not really a good fit for today's world of privacy regulations, where deletion of data should be possible and guaranteed.
  - lgas 4y ago
    Datomic addresses this via Excision. https://docs.datomic.com/on-prem/reference/excision.html https://docs.datomic.com/on-prem/reference/excision.html
  - _rcil 4y ago
    XTDB has a simple `evict` operation to resolve precisely this issue: https://docs.xtdb.com/language-reference/datalog-transactions/#evict https://docs.xtdb.com/language-reference/datalog-transaction...
  - john567 4y ago
    This can be solved in different ways. First, everything is an entity, so you can enrich attributes with information about how they should manage PII/SPI data. There's nothing built-in but you can build meta models easily. Second, if you really have stringent requirements you would encrypt the data that is to be protected and throw away the encryption keys when the data needs to be purged. Datomic does allow you to opt out of history so the keys would not be recoverable in that case. Or, you just excise the data, at which point the information will be purged from the database. It's not instant, index rebuilds and garbage collection need to take place before it's really gone. You're in control. The fact that you have history isn't a problem.

Mostly when you need a ledger like functionality to your data. Datomic preserves the history of all the changes by design so it is very easy to reconstruct the history of changes to your data. If this describes the majority of your problem space then Datomic is a good fit. Otherwise I would stick to a popular RDBMS like Postgres to leverage the community behind it. You will have a much easier time interacting with a well understood open source engine.

emccue 4y ago
I think this is true - you will have an easier time not blazing new trails in general - but the need of ledger-like functionality isn't the distinguishing factor. Preserving the history of all changes isn't just for reconstructing that history later, it's a central component in maintaining ACID while allowing for distributed readers. Datomic has a singular transactor - node that can accept writes. So all transactions happen in sequence and consistency is maintained within the data model. So your database goes DB (0 transactions) -> DB' (1 transaction committed) -> DB'' (2 transactions committed) With normal databases, when DB'' is created, DB' is lost. You change stuff in place. This means that anyone wanting to observe a consistent view of the database needs to coordinate with writes. This is why SQL dbs are single giant machines. For Cassandra, Dynamo, Mongo, etc you still change stuff in place, but you just accept that things will be potentially inconsistent ("eventually consistent") for your readers. Once you drop that part of ACID you can scale your readers and your writers arbitrarily. With Datomic DB, DB', and DB'' are still maintained. If you want to read the database "now" you can get a handle to the state that your reader thinks is the most recent. This might not be behind the absolute newest state that the transactor knows about, but it is still a consistent view of the database. Its because of this you can scale your read nodes independently of the system. Because if you are reading DB' you know the value of that won't change you can load chunks of the indexes into the memory for caching and even embed the querying into the application making queries. The proprietary-ness and JVM-ness of Datomic are its biggest weaknesses. I choose to believe those weaknesses can be addressed.
- huahaiy 4y ago
  Every database that implements MVCC does this, that means most ACID databases. So "reading a consistent view" is not unique for Datomic.
- Nihilartikel 4y ago
  I'm waiting to have a need to use it, but XTDB is a nice fully open bitemporal 'lite' Datomic that I have been keeping an eye on.
- runekaagaard 4y ago
  Ah yes, makes me remember the amazing talk by Rich Hickey called "The database as a Value" [1]. I felt like my brain was reorganized after watching it. The simplicity of Datomic is very appealing, it's just a timestamped series of EAV entries, and some fancy indexing. Avoiding the latency (the database over there issue) sounds like a big win. Most of the simplicity is made possible by only having one single synchronous writer which is a major drawback you have to be able to live with. Some write-heavy applications would struggle. Related to the timetravel feature I've bookmarked a blog post by Valentin Waeselynck where he talks about some of the misconceptions and drawbacks. [2]. My company have a not super-great setup with triggers on MySQL, and have longed for MariaDBs SQL 2011 compliant Temporal Tables which looks very interesting [3]. I guess an advantage with Datomic is that you don't have to create an entire table history row when all you want is to update a single field. I was interested enough in Datomic that I started building a Python bridge but never finished it [4]. An added bonus was that I learned about the Transit serialization format which was ALSO super inspiring [5]. [1] https://www.youtube.com/watch?v=D6nYfttnVco [2] https://vvvvalvalval.github.io/posts/2017-07-08-Datomic-this-is-not-the-history-youre-looking-for.html [3] https://mariadb.com/resources/blog/temporal-tables-part-1/ [4] https://github.com/runekaagaard/brygge/blob/master/src/jolle/example.py [5] https://cognitect.com/blog/2014/7/22/transit
john567 4y ago
This is true but it's a false premise. You have history but it's clunky to work with. You won't be able to "source" and recreate the past without some work and unexpected things will happen when you "source" the past with the most recent code. It useful but the history aspect is easily misunderstood.
- emccue 4y ago
  In this respect XTDB has a more directly useful model
- blain_the_train 4y ago
  Can you explain what "source" means?
  - john567 4y ago
    Oh, yeah, think of event sourcing. You have a series of events leading up (aggregate) into some state. This can be done in different ways but with Datomic you can request a point in time database. This database doesn't exist so it needs to be "sourced" from history. This means running through everything that happened up to some point to create a snapshot of the database state (or query).

emccue 4y ago

As the world stands today, it makes sense to use Datomic if you are writing code on the JVM and are willing to pay for the database and associated support. In a hypothetical future world where the data model and query approaches have more widespread open source adoption it is competitive for a lot of the same use cases.

casion 4y ago

In addition to what was said, Datomic Cloud automates a lot of the annoying and difficult to suss deployment for application dev, not just the database. You can deploy large scalable apps in minutes and easily deploy new code in minutes with no down time (or architecting a deployment system yourself).

alecco 4y ago

Datomic is elegant and it has some interesting properties like temporal. This is a huge win for many cases where business logic needs them (e.g. finance). I think it would make more sense for clients also running JVM. I don't have much experience with it but it looks very good. -- a SQL engine dev

john567 4y ago

What I've find most useful is the immutable nature of the database. That within a reactive framework for UX is great combination. You get really cool things to happen with little effort. Also, because the database is a value. You can easily do mock data and isolated transactions locally to experiment and test. This makes some things easier. Another cool thing is how you do cross database queries by simply passing multiple databases to a function. It will pull in the subset of the dataset needed to compute the query result. It's that easy to do. It's not good at dealing with high volume transactions (a lot of writes) it's not too bad but it's not built for that primarily.

amelius 4y ago
> What I've find most useful is the immutable nature of the database. Not really a good fit for today's world of privacy regulations, where deletion of data should be possible and guaranteed.
- lgas 4y ago
  Datomic addresses this via Excision. https://docs.datomic.com/on-prem/reference/excision.html https://docs.datomic.com/on-prem/reference/excision.html
- _rcil 4y ago
  XTDB has a simple `evict` operation to resolve precisely this issue: https://docs.xtdb.com/language-reference/datalog-transactions/#evict https://docs.xtdb.com/language-reference/datalog-transaction...
- john567 4y ago
  This can be solved in different ways. First, everything is an entity, so you can enrich attributes with information about how they should manage PII/SPI data. There's nothing built-in but you can build meta models easily. Second, if you really have stringent requirements you would encrypt the data that is to be protected and throw away the encryption keys when the data needs to be purged. Datomic does allow you to opt out of history so the keys would not be recoverable in that case. Or, you just excise the data, at which point the information will be purged from the database. It's not instant, index rebuilds and garbage collection need to take place before it's really gone. You're in control. The fact that you have history isn't a problem.