5 ms·

Thinking in Datomic

96 points by pelle 14y ago

mbreese 14y ago
Couldn't you just model your data in a traditional RDBMS with a time stamp as well? In fact for mutable data that you'd like to keep old versions of, this is pretty standard. A simple design would be to have a separate table for person_locations that mapped a person to a location. With everything else, the standard RDBMS table could be considered as having a 'snapshot' of the Datomic values. I'm still not sure what benefit this has over a traditional DB. Perhaps I'll just have to wait for the next post.
- jcoene 14y ago
  Of course you can make something similar if you're a) willing to build it and 2) don't care about the tremendous amount of complexity you're adding to your application in order to do so. Both of those sound like bad deals if there's a packaged solution like Datomic that's built specifically for the use case.
- pelle 14y ago
  You can do it all yourself if you want, but Datomic does it for you without you having to do anything. Also as it does it on a datom level it is a lot more efficient than versioned rows. Interestingly one of the early selling points of Postgresql was this time travel functionality. But it was yanked out in 6.2 http://www.postgresql.org/docs/6.3/interactive/c0503.htm http://www.postgresql.org/docs/6.3/interactive/c0503.htm
  - mbreese 14y ago
    Yes, but you also loose the flexibility of not doing it. I'd rather work in an established RDMBS and configure in the design I'd like to use rather than use a database that requires a specific configuration. I'm sure that Datomic has it's place, but the examples you gave in this post aren't that convincing.
    - pelle 14y ago
      I agree with you this is not on its own necessarily a good reason to switch to Datomic. I will be getting into more detailed examples later. My real point with this post was more to talk about how to model the data using datoms and not specifically the temporal aspects of it. I made many mistakes in my original data models in datomic based on many years of rdbms thinking.
- mattdeboard 14y ago
  The difference between Datomic and RDBMS isn't JUST a timestamp FYI. In fact I'd expect/hope this blog post is a just a prelude to Datomic's transactional model which is the real differentiator.
  - pelle 14y ago
    Absolutely. That is where I'm going in the next article. The transactions are extremely cool.
    - kinleyd 14y ago
      I look forward to it.
- snorkel 14y ago
  Yes, you can do similar in a traditional RDMS, you have a table called Person and another table called Person_properties with a schema of pid, property, value, timestamp. The object model for Person would have a save() and load() method that knows which properties belong to the base table (base properties) and which properties spill over in into the triple table. The separate properties table has the convenience of stashing arbitrary persistent properties in a Person without needing to migrate the underlying schema or change the base definition.
mtrn 14y ago
The temporal aspect seems to be somewhat ignored by mainstream db development - in some fields, e.g. BI you'll have slowly changing dimensions; and the event sourcing pattern promises a time-machine view on data. Since I'm writing a history-aware application at the moment, I recently looked into different patterns for this and trying a mixed strategy at the moment (SQL DB used as document store and a single event log, that accumulates changes - a lean approach though, a few hundred lines python for the data access layer; what always gets ugly is the validation, which your application must take care of). I wish, there was more hands-on material on the subject (some resources dive depth into bi-temporal modeling, but I feel your schemas can get complex (= expensive) very fast).
- dustingetz 14y ago
  by my understanding, datomic's persistence[1] layer is just a big huge persistent[2] data structure. there is tons of literature, books, papers, etc about persistent data structures in the functional programming community. [1][2] 'persistent' means different things here.
  - calebmpeterson 14y ago
    So, Datomic is a persistent data structure persisted to nonvolatile storage... I've wanted one of these for at least the last 2 years. If only there was an open source implementation of this concept that I could run on my own hardware. Does anyone know of such a beast? And, no, I don't want to roll my own versioned/timestamped row schema in an RDBMS - been there, suffered that.
    - RehnoLindeque 14y ago
      Acid-State for haskell (http://acid-state.seize.it/ http://acid-state.seize.it/) is based on a similar ideology and in the public domain. (Oh, but minus the time dimension I'm afraid.)
      - DanWaterworth 14y ago
jerf 14y ago
There have been numerous triple stores. Mozilla/Firefox even used one as its core backend for a while before ripping it out and replacing it with SQLite. Why do you think those failed, and how is Datomic going to avoid the same failures? That is, I've seen this pitched as the solution to all our data woes numerous times now, why is this time different?
- deleted 14y ago
  [deleted]
- mtrimpe 14y ago
  It's not a triple store. It's closer to an OODBMS focused on persisting large object graphs with an immutable append-only datastructure, thus maintaining full history and being quite scalable with a single-writer many-reader configuration.
  - pelle 14y ago
    It is actually implemented more like a triple store than an oodbms. Just with time added as a 4th parameter. Their Entity class makes it feel a bit like an oodbms though in that it kind of acts like a hash map.
    - mtrimpe 14y ago
      Could you explain how it's more like a triple store with a time parameter? My previous explanation already papered over a ton of fundamental differences in order to make a somewhat understandable generalization, but I can't make the jump to triple-store from my understanding of Datomic. I basically see it as "just a single-writer many-reader distributed fully persistent data-structure."
      - jerf 14y ago
        I'm sort of sorry to turn this around on you, because it's generally poor form, but given the contents of the linked article which describes Datomic exclusively in the form of timestamped triples, can you explain how it's not a triple store? If it's because you store objects as many triples with the same subject and different adjectives... err, well, yes, that's how you store objects in a triple store. Not remotely new, and still subject to my initial question.
damncabbage 14y ago
"Datomic is so different than regular databases that your average developer will probably chose to ignore it." Playing to the Well I'm obviously above average gut feeling. Cute.

96 points by pelle 14y ago

Couldn't you just model your data in a traditional RDBMS with a time stamp as well? In fact for mutable data that you'd like to keep old versions of, this is pretty standard. A simple design would be to have a separate table for person_locations that mapped a person to a location. With everything else, the standard RDBMS table could be considered as having a 'snapshot' of the Datomic values. I'm still not sure what benefit this has over a traditional DB. Perhaps I'll just have to wait for the next post.

jcoene 14y ago
Of course you can make something similar if you're a) willing to build it and 2) don't care about the tremendous amount of complexity you're adding to your application in order to do so. Both of those sound like bad deals if there's a packaged solution like Datomic that's built specifically for the use case.
pelle 14y ago
You can do it all yourself if you want, but Datomic does it for you without you having to do anything. Also as it does it on a datom level it is a lot more efficient than versioned rows. Interestingly one of the early selling points of Postgresql was this time travel functionality. But it was yanked out in 6.2 http://www.postgresql.org/docs/6.3/interactive/c0503.htm http://www.postgresql.org/docs/6.3/interactive/c0503.htm
- mbreese 14y ago
  Yes, but you also loose the flexibility of not doing it. I'd rather work in an established RDMBS and configure in the design I'd like to use rather than use a database that requires a specific configuration. I'm sure that Datomic has it's place, but the examples you gave in this post aren't that convincing.
  - pelle 14y ago
    I agree with you this is not on its own necessarily a good reason to switch to Datomic. I will be getting into more detailed examples later. My real point with this post was more to talk about how to model the data using datoms and not specifically the temporal aspects of it. I made many mistakes in my original data models in datomic based on many years of rdbms thinking.
mattdeboard 14y ago
The difference between Datomic and RDBMS isn't JUST a timestamp FYI. In fact I'd expect/hope this blog post is a just a prelude to Datomic's transactional model which is the real differentiator.
- pelle 14y ago
  Absolutely. That is where I'm going in the next article. The transactions are extremely cool.
  - kinleyd 14y ago
    I look forward to it.
snorkel 14y ago
Yes, you can do similar in a traditional RDMS, you have a table called Person and another table called Person_properties with a schema of pid, property, value, timestamp. The object model for Person would have a save() and load() method that knows which properties belong to the base table (base properties) and which properties spill over in into the triple table. The separate properties table has the convenience of stashing arbitrary persistent properties in a Person without needing to migrate the underlying schema or change the base definition.

mtrn 14y ago

The temporal aspect seems to be somewhat ignored by mainstream db development - in some fields, e.g. BI you'll have slowly changing dimensions; and the event sourcing pattern promises a time-machine view on data. Since I'm writing a history-aware application at the moment, I recently looked into different patterns for this and trying a mixed strategy at the moment (SQL DB used as document store and a single event log, that accumulates changes - a lean approach though, a few hundred lines python for the data access layer; what always gets ugly is the validation, which your application must take care of). I wish, there was more hands-on material on the subject (some resources dive depth into bi-temporal modeling, but I feel your schemas can get complex (= expensive) very fast).

dustingetz 14y ago
by my understanding, datomic's persistence[1] layer is just a big huge persistent[2] data structure. there is tons of literature, books, papers, etc about persistent data structures in the functional programming community. [1][2] 'persistent' means different things here.
- calebmpeterson 14y ago
  So, Datomic is a persistent data structure persisted to nonvolatile storage... I've wanted one of these for at least the last 2 years. If only there was an open source implementation of this concept that I could run on my own hardware. Does anyone know of such a beast? And, no, I don't want to roll my own versioned/timestamped row schema in an RDBMS - been there, suffered that.
  - RehnoLindeque 14y ago
    Acid-State for haskell (http://acid-state.seize.it/ http://acid-state.seize.it/) is based on a similar ideology and in the public domain. (Oh, but minus the time dimension I'm afraid.)
    - DanWaterworth 14y ago

jerf 14y ago

There have been numerous triple stores. Mozilla/Firefox even used one as its core backend for a while before ripping it out and replacing it with SQLite. Why do you think those failed, and how is Datomic going to avoid the same failures? That is, I've seen this pitched as the solution to all our data woes numerous times now, why is this time different?

deleted 14y ago
[deleted]
mtrimpe 14y ago
It's not a triple store. It's closer to an OODBMS focused on persisting large object graphs with an immutable append-only datastructure, thus maintaining full history and being quite scalable with a single-writer many-reader configuration.
- pelle 14y ago
  It is actually implemented more like a triple store than an oodbms. Just with time added as a 4th parameter. Their Entity class makes it feel a bit like an oodbms though in that it kind of acts like a hash map.
  - mtrimpe 14y ago
    Could you explain how it's more like a triple store with a time parameter? My previous explanation already papered over a ton of fundamental differences in order to make a somewhat understandable generalization, but I can't make the jump to triple-store from my understanding of Datomic. I basically see it as "just a single-writer many-reader distributed fully persistent data-structure."
    - jerf 14y ago
      I'm sort of sorry to turn this around on you, because it's generally poor form, but given the contents of the linked article which describes Datomic exclusively in the form of timestamped triples, can you explain how it's not a triple store? If it's because you store objects as many triples with the same subject and different adjectives... err, well, yes, that's how you store objects in a triple store. Not remotely new, and still subject to my initial question.

damncabbage 14y ago

"Datomic is so different than regular databases that your average developer will probably chose to ignore it." Playing to the Well I'm obviously above average gut feeling. Cute.