9 ms·

Unofficial Guide to Datomic Internals (2014)

145 points by grzm 6y ago

vlmutolo 6y ago
this theme is hilarious and infuriating EDIT: You can comment out the yellow background image in the style editor and it becomes something reasonable
- kgwxd 6y ago
  Too late. Read the whole thing before coming back here to see your suggestion, now everything here, and everywhere else, is yellowish, no matter what I set in the style editor.
- adamkl 6y ago
  I quite like the black-on-yellow theme. It’s different, and distinctive (I’ve read a few things on tonsky’s site, so I know what to expect). Even more hilarious is switching to “dark mode”.
- ddlutz 6y ago
  It gave me a migraine reading half of the article, couldn't continue :(
dgb23 6y ago
Additional details about Datoms: To determine whether a Datom is being rectracted or added there is a fifth element in the tuple [0]. There are many similarities to modelling temporal data in SQL [1]. But Datoms are simpler and more open as you can freely build relations between them (composable), similar to a graph-db. [0] https://docs.datomic.com/cloud/whatis/data-model.html https://docs.datomic.com/cloud/whatis/data-model.html [1] https://en.wikipedia.org/wiki/Temporal_database https://en.wikipedia.org/wiki/Temporal_database
archarios 6y ago
I went to a Clojure meetup one time and they all went on about how using Datomic in production is a nightmare and it's generally an over-engineered product that isn't worth the trouble in the end. Do most people who have dealt with Datomic in production feel this way?
- dwohnitmok 6y ago
  Anecdotally I know of one company which is also in the same boat and generally regrets their usage of Datomic and is trying to move away from it last I talked with them. However, there's also people on HN like dustingetz who have had a great time with Datomic and use it as a core component of their product. I just wish Cognitect would allow people to run public benchmarks of Datomic to make it easier to evaluate its tradeoffs.
  - lukashrb 6y ago
    Do you have more detailed info on this? I guess this could really help making a decision and understanding the tradeoff's of using datomic.
    - dwohnitmok 6y ago
      What the company ran into? Unfortunately not :/. It was a quick chat in an informal setting with their VP of engineering (I think?) that really was just a "huh, interesting moment" for me (although I've coded in Clojure for a full-time job before I have essentially no personal experience with Datomic). As for the positive side, I think dustingetz monitors Clojure and Datomic threads pretty closely so maybe they can chime in here.
  - zeroDivisible 6y ago
    What is the policy of Cognitect re: public benchmarks? I did not know that.
    - dwohnitmok 6y ago
      > The Licensee hereby agrees, without the prior written consent of Cognitect, which may be withheld or conditioned at Cognitect’s sole discretion, it will not... publicly display or communicate the results of internal performance testing or other benchmarking or performance evaluation of the Software From the Datomic EULA here: https://www.datomic.com/on-prem-eula.html https://www.datomic.com/on-prem-eula.html
twic 6y ago
> This simplicity enables Datomic to do more than any relational DB or KV storage can ever afford > Datomic does not manage persistence itself, instead, it outsources storage problems to databases implemented by other people. Data can be kept, at your expense, in DynamoDB, Riak, Infinispan, Couchbase or SQL database. These things can't both be true.
- reilly3000 6y ago
  I think in this context "do more" refers to querying capabilities than the costs of running the system.
- adamkl 6y ago
  Why not? It really depends on the definition of “do more”. From what I understand, Datomic’s model is far more flexible than many other databases, and has built in time travelling capability due to its accretion of immutable data. It’s architecture does, in fact, allow you to choose the storage provider, and it’s considered an external concern. Those are very compelling reasons to use it, but part of the trade off is write-scalability, and potentially, raw performance. So maybe it’s “do more” within certain limitations (which is up to you to decide if those limitations are a deal breaker). Here’s a great talk about Datomic’s architecture for more details: https://youtu.be/9TYfcyvSpEQ https://youtu.be/9TYfcyvSpEQ
- tnisonoff 6y ago
  If the argument is Datomic can't be simpler than a relational DB because it can utilize it for persistence, then you'd have to argue that a relational DB can't be simpler than directly using a hard drive for your storage solution.
  - jayd16 6y ago
    "This simplicity enables a relation DB to do more than any hard drive can ever afford." That also seems debatable for the exact same reasons. I don't think this would convince anyone that needs convincing.
  - deleted 6y ago
    [deleted]
paulgb 6y ago
One thing I've never understood is why all the indexes have transaction last. One of the selling points of Datomic is that it supports as-of queries, but using the EAVT or AEVT indexes requires it to scan all historic values of that attribute, right? In most situations this is probably fine, but if you have data that changes frequently it seems like this could slow queries down compared to an EATV or AETV index. It's also likely that the people who made Datomic are both smarter about this stuff than me and put more thought into it than I have, so I'd love to know what the reasoning behind the choice of index is. (PS @dang it would be nice to have (2014) in the title)
- deleted 6y ago
  [deleted]
- nlitened 6y ago
  The page mentions the Log index which is sorted by transaction id. It should be enough to support as-of if I understand correctly.
  - paulgb 6y ago
    The log index supports as-of if you know the actual transaction ID, but if you want to look up by entity/attribute efficiently it's not much help because you don't know when the data point you're interested in was last modified.
    - nlitened 6y ago
      I think in this case you’d find all datoms via normal EAVT index and then sort the results by transaction id, dropping everything after your desired transaction.
- lukashrb 6y ago
  Not necessarily. Take a look at this implementation https://aosabook.org/en/500L/an-archaeology-inspired-database.html https://aosabook.org/en/500L/an-archaeology-inspired-databas... There you can only retrieve the top layer and don't have to scan all historic data, it's only in-memory though.
jmiskovic 6y ago
For uninitiated, Nikita has created in-memory database with mostly compatible API (datalog) for ClojureScript or Javascript. My impression is that his variant actually has more wide-spread use than original Datomic, based on number of open source projects that use each. I used DataScript for a while to get familiar with graph database querying. I was fascinated how easy it is to construct queries that mine obscure relations between distantly related entities. I hope I get to use similar tech again.
JimmyRuska 6y ago
We used datomic in production at Time Inc around 2016. The idea of an immutable database where you can track changes over time, or query the state of the universe at any given point, sounded amazing for marketing or compliance use cases. Unfortunately from a dev standpoint it did not feel like mature system, and the performance was not where we needed it to be. Probably the most advanced database for triple stores these days is RDFox ( https://www.youtube.com/watch?v=-DnmuHtywFs https://www.youtube.com/watch?v=-DnmuHtywFs ). While datomic uses datalog for querying, RDFox uses datalog for database reasoning, and sparql, a w3 standard for querying. As you add data to the database you can infer new facts. If you want immutability, simply add data in append mode only with a timestamp. But this idea you can add the business rules/logic to the database, and have it incrementally apply that logic as you add data is a recent advance by oxford AI research.
- kendallgclark 6y ago
  Stardog is full of features, performance, and enterprise hardening Rdfox hasn’t even started thinking about yet.

145 points by grzm 6y ago

this theme is hilarious and infuriating EDIT: You can comment out the yellow background image in the style editor and it becomes something reasonable

kgwxd 6y ago
Too late. Read the whole thing before coming back here to see your suggestion, now everything here, and everywhere else, is yellowish, no matter what I set in the style editor.
adamkl 6y ago
I quite like the black-on-yellow theme. It’s different, and distinctive (I’ve read a few things on tonsky’s site, so I know what to expect). Even more hilarious is switching to “dark mode”.
ddlutz 6y ago
It gave me a migraine reading half of the article, couldn't continue :(

dgb23 6y ago

Additional details about Datoms: To determine whether a Datom is being rectracted or added there is a fifth element in the tuple [0]. There are many similarities to modelling temporal data in SQL [1]. But Datoms are simpler and more open as you can freely build relations between them (composable), similar to a graph-db. [0] https://docs.datomic.com/cloud/whatis/data-model.html https://docs.datomic.com/cloud/whatis/data-model.html [1] https://en.wikipedia.org/wiki/Temporal_database https://en.wikipedia.org/wiki/Temporal_database

archarios 6y ago

I went to a Clojure meetup one time and they all went on about how using Datomic in production is a nightmare and it's generally an over-engineered product that isn't worth the trouble in the end. Do most people who have dealt with Datomic in production feel this way?

dwohnitmok 6y ago
Anecdotally I know of one company which is also in the same boat and generally regrets their usage of Datomic and is trying to move away from it last I talked with them. However, there's also people on HN like dustingetz who have had a great time with Datomic and use it as a core component of their product. I just wish Cognitect would allow people to run public benchmarks of Datomic to make it easier to evaluate its tradeoffs.
- lukashrb 6y ago
  Do you have more detailed info on this? I guess this could really help making a decision and understanding the tradeoff's of using datomic.
  - dwohnitmok 6y ago
    What the company ran into? Unfortunately not :/. It was a quick chat in an informal setting with their VP of engineering (I think?) that really was just a "huh, interesting moment" for me (although I've coded in Clojure for a full-time job before I have essentially no personal experience with Datomic). As for the positive side, I think dustingetz monitors Clojure and Datomic threads pretty closely so maybe they can chime in here.
- zeroDivisible 6y ago
  What is the policy of Cognitect re: public benchmarks? I did not know that.
  - dwohnitmok 6y ago
    > The Licensee hereby agrees, without the prior written consent of Cognitect, which may be withheld or conditioned at Cognitect’s sole discretion, it will not... publicly display or communicate the results of internal performance testing or other benchmarking or performance evaluation of the Software From the Datomic EULA here: https://www.datomic.com/on-prem-eula.html https://www.datomic.com/on-prem-eula.html

twic 6y ago

> This simplicity enables Datomic to do more than any relational DB or KV storage can ever afford > Datomic does not manage persistence itself, instead, it outsources storage problems to databases implemented by other people. Data can be kept, at your expense, in DynamoDB, Riak, Infinispan, Couchbase or SQL database. These things can't both be true.

reilly3000 6y ago
I think in this context "do more" refers to querying capabilities than the costs of running the system.
adamkl 6y ago
Why not? It really depends on the definition of “do more”. From what I understand, Datomic’s model is far more flexible than many other databases, and has built in time travelling capability due to its accretion of immutable data. It’s architecture does, in fact, allow you to choose the storage provider, and it’s considered an external concern. Those are very compelling reasons to use it, but part of the trade off is write-scalability, and potentially, raw performance. So maybe it’s “do more” within certain limitations (which is up to you to decide if those limitations are a deal breaker). Here’s a great talk about Datomic’s architecture for more details: https://youtu.be/9TYfcyvSpEQ https://youtu.be/9TYfcyvSpEQ
tnisonoff 6y ago
If the argument is Datomic can't be simpler than a relational DB because it can utilize it for persistence, then you'd have to argue that a relational DB can't be simpler than directly using a hard drive for your storage solution.
- jayd16 6y ago
  "This simplicity enables a relation DB to do more than any hard drive can ever afford." That also seems debatable for the exact same reasons. I don't think this would convince anyone that needs convincing.
- deleted 6y ago
  [deleted]

paulgb 6y ago

One thing I've never understood is why all the indexes have transaction last. One of the selling points of Datomic is that it supports as-of queries, but using the EAVT or AEVT indexes requires it to scan all historic values of that attribute, right? In most situations this is probably fine, but if you have data that changes frequently it seems like this could slow queries down compared to an EATV or AETV index. It's also likely that the people who made Datomic are both smarter about this stuff than me and put more thought into it than I have, so I'd love to know what the reasoning behind the choice of index is. (PS @dang it would be nice to have (2014) in the title)

deleted 6y ago
[deleted]
nlitened 6y ago
The page mentions the Log index which is sorted by transaction id. It should be enough to support as-of if I understand correctly.
- paulgb 6y ago
  The log index supports as-of if you know the actual transaction ID, but if you want to look up by entity/attribute efficiently it's not much help because you don't know when the data point you're interested in was last modified.
  - nlitened 6y ago
    I think in this case you’d find all datoms via normal EAVT index and then sort the results by transaction id, dropping everything after your desired transaction.
lukashrb 6y ago
Not necessarily. Take a look at this implementation https://aosabook.org/en/500L/an-archaeology-inspired-database.html https://aosabook.org/en/500L/an-archaeology-inspired-databas... There you can only retrieve the top layer and don't have to scan all historic data, it's only in-memory though.

jmiskovic 6y ago

For uninitiated, Nikita has created in-memory database with mostly compatible API (datalog) for ClojureScript or Javascript. My impression is that his variant actually has more wide-spread use than original Datomic, based on number of open source projects that use each. I used DataScript for a while to get familiar with graph database querying. I was fascinated how easy it is to construct queries that mine obscure relations between distantly related entities. I hope I get to use similar tech again.

JimmyRuska 6y ago

We used datomic in production at Time Inc around 2016. The idea of an immutable database where you can track changes over time, or query the state of the universe at any given point, sounded amazing for marketing or compliance use cases. Unfortunately from a dev standpoint it did not feel like mature system, and the performance was not where we needed it to be. Probably the most advanced database for triple stores these days is RDFox ( https://www.youtube.com/watch?v=-DnmuHtywFs https://www.youtube.com/watch?v=-DnmuHtywFs ). While datomic uses datalog for querying, RDFox uses datalog for database reasoning, and sparql, a w3 standard for querying. As you add data to the database you can infer new facts. If you want immutability, simply add data in append mode only with a timestamp. But this idea you can add the business rules/logic to the database, and have it incrementally apply that logic as you add data is a recent advance by oxford AI research.

kendallgclark 6y ago
Stardog is full of features, performance, and enterprise hardening Rdfox hasn’t even started thinking about yet.