5 ms·

The Architecture of Datomic

89 points by chillax 14y ago

jeremyjh 14y ago
Datomic is intriguing and powerful but I'd be reluctant to use a proprietary database developed by such a small company. I think Riak has a better model for free/paid usage. There is nothing like Datomic but there are so many excellent open-source databases these days.
- batgaijin 14y ago
  How long do they plan on keeping it proprietary? Is there a timeline?
  - fogus 14y ago
    I doubt very much that there is a timeline. It's a product and the last I heard was staying close sourced. Having said that, I would not be surprised if some of its libraries are open sourced. I know of at least one that is in the pipeline.
    - tom_b 14y ago
      If you say the datalog query aspect of Datomic, I would be tremendously (maybe embarrassingly) happy.
    - tbatchelli 14y ago
      Datalog for clojure's data structures would take Clojure to a new productivity level. It would also greatly simplify a lot of the code out there today, further driving software to be more data-oriented. Data is the ultimate API. Please make it be Datalog.
      - richhickey 14y ago
        Datomic Free gives you that capability right now, for free, no strings attached, and no need to use a Datomic db. Not open source, but free as in gratis, a la Creative Commons. We may trim down the deps a bit for that use case.
        tbatchelli 14y ago
        A trimmed down version is great news. Datalog makes Clojure so much better! Ideally it would be part of Clojure itself, but I'll take what I can get :)
        SnootyMonkey 14y ago
        I would also love to see a trimmed down version just to use datalog.
      - fogus 14y ago
        The datomic-free jar provides Datalog. I use it myself https://clojars.org/com.datomic/datomic-free https://clojars.org/com.datomic/datomic-free.
      - jamii 14y ago
        I'm working on a port of BloomL for clojure which will include a semi-naive datalog interpreter (it will already support naive, monotonic datalog by tonight). It will likely never be as well optimised as the Datomic libs but it is LGPL. Source: https://github.com/jamii/droplet https://github.com/jamii/droplet Background: http://www.bloom-lang.net/ http://www.bloom-lang.net/ http://boom.cs.berkeley.edu/ http://boom.cs.berkeley.edu/ http://db.cs.berkeley.edu/papers/UCB-lattice-tr.pdf http://db.cs.berkeley.edu/papers/UCB-lattice-tr.pdf
- robertfw 14y ago
  I agree completely; we are looking at tech choices for a new project and Datomic is an excellent fit for the way we want our systems to work, but we're just not comfortable with the closed/paid model, especially for such a young product. For now we are looking at ways to use an accumulation based system (vs update-in-place) in PostgreSQL. If anyone has any experience with this I'd love to hear about it!
  - rikf 14y ago
    Take a look at event sourcing. This is a good introduction. http://martinfowler.com/eaaDev/EventSourcing.html http://martinfowler.com/eaaDev/EventSourcing.html
  - dm3 14y ago
    There is another database which can be used in the 'accumulation-based' space: http://geteventstore.com http://geteventstore.com It has the same main drawbacks as Datomic - it's young and multi-node deployment comes with commercial plan only, however it's really fast and easy to work with once you embrace the event-sourcing programming model.
- benbjohnson 14y ago
  I'm the dev for an open source event-based database called Sky (http://skydb.io http://skydb.io). Similar data model but it's aimed at a different use case than Datomic -- it's primarily used for analytics. The event data model is nice for avoiding conflicts but it's an up road battle to try and replace traditional relational databases. I find that storing historical events is primarily useful for either analytics or auditing. It makes more sense to move that off your primary data store if possible. I do find that making databases smarter about the data they hold can really provide some great optimizations. Sky can crunch through about 60MM events per second on a single core because it is built for event data models. I'm sure Datomic can probably get some good optimizations too. I'll be curious to see some benchmarks. Here's a demo app of Sky crunching all the events in the GitHub Archive (starts around 3m:20s): http://skydb.io/blog/github-archive-visualizer.html http://skydb.io/blog/github-archive-visualizer.html I'm working on multi-core distribution right now and I'll be doing multi-node after that. Check out the web site or GitHub (https://github.com/skydb/sky https://github.com/skydb/sky) for more information. It's MIT licensed so feel free to hack away.
  - fizx 14y ago
    You really need a "how it works" section. Few people are willing to read the source code to see if you have interesting or useful ideas.
    - benbjohnson 14y ago
      That's a good point. I'm cleaning up and stabilizing the code right now and part of my v0.2.0 release is going to be documentation. I did a presentation recently about some of the ideas behind the database: https://speakerdeck.com/benbjohnson/behavioral-databases https://speakerdeck.com/benbjohnson/behavioral-databases The intro on the GitHub Archive Visualizer video explains some more of it too: https://vimeo.com/51629936 https://vimeo.com/51629936 I'll add that stuff to the README though.
- chillax 14y ago
  In related news Riak was in fact added as supported storage today: http://blog.datomic.com/2012/11/riak-and-couchbase-support.html http://blog.datomic.com/2012/11/riak-and-couchbase-support.h...
mvalle 14y ago
Sorry for going off-topic, but is there any explanation why InfoQ does not have an entry on Wikipedia? Is it just because no one has bothered to do it yet, or is there some other reason why this quite big, and frequently cited, site is not on Wikipedia?
krosaen 14y ago
Related: presentation on how datomic was written in clojure http://www.infoq.com/presentations/Datomic http://www.infoq.com/presentations/Datomic

89 points by chillax 14y ago

Datomic is intriguing and powerful but I'd be reluctant to use a proprietary database developed by such a small company. I think Riak has a better model for free/paid usage. There is nothing like Datomic but there are so many excellent open-source databases these days.

batgaijin 14y ago
How long do they plan on keeping it proprietary? Is there a timeline?
- fogus 14y ago
  I doubt very much that there is a timeline. It's a product and the last I heard was staying close sourced. Having said that, I would not be surprised if some of its libraries are open sourced. I know of at least one that is in the pipeline.
  - tom_b 14y ago
    If you say the datalog query aspect of Datomic, I would be tremendously (maybe embarrassingly) happy.
  - tbatchelli 14y ago
    Datalog for clojure's data structures would take Clojure to a new productivity level. It would also greatly simplify a lot of the code out there today, further driving software to be more data-oriented. Data is the ultimate API. Please make it be Datalog.
    - richhickey 14y ago
      Datomic Free gives you that capability right now, for free, no strings attached, and no need to use a Datomic db. Not open source, but free as in gratis, a la Creative Commons. We may trim down the deps a bit for that use case.
      - tbatchelli 14y ago
        A trimmed down version is great news. Datalog makes Clojure so much better! Ideally it would be part of Clojure itself, but I'll take what I can get :)
      - SnootyMonkey 14y ago
        I would also love to see a trimmed down version just to use datalog.
    - fogus 14y ago
      The datomic-free jar provides Datalog. I use it myself https://clojars.org/com.datomic/datomic-free https://clojars.org/com.datomic/datomic-free.
    - jamii 14y ago
      I'm working on a port of BloomL for clojure which will include a semi-naive datalog interpreter (it will already support naive, monotonic datalog by tonight). It will likely never be as well optimised as the Datomic libs but it is LGPL. Source: https://github.com/jamii/droplet https://github.com/jamii/droplet Background: http://www.bloom-lang.net/ http://www.bloom-lang.net/ http://boom.cs.berkeley.edu/ http://boom.cs.berkeley.edu/ http://db.cs.berkeley.edu/papers/UCB-lattice-tr.pdf http://db.cs.berkeley.edu/papers/UCB-lattice-tr.pdf
robertfw 14y ago
I agree completely; we are looking at tech choices for a new project and Datomic is an excellent fit for the way we want our systems to work, but we're just not comfortable with the closed/paid model, especially for such a young product. For now we are looking at ways to use an accumulation based system (vs update-in-place) in PostgreSQL. If anyone has any experience with this I'd love to hear about it!
- rikf 14y ago
  Take a look at event sourcing. This is a good introduction. http://martinfowler.com/eaaDev/EventSourcing.html http://martinfowler.com/eaaDev/EventSourcing.html
- dm3 14y ago
  There is another database which can be used in the 'accumulation-based' space: http://geteventstore.com http://geteventstore.com It has the same main drawbacks as Datomic - it's young and multi-node deployment comes with commercial plan only, however it's really fast and easy to work with once you embrace the event-sourcing programming model.
benbjohnson 14y ago
I'm the dev for an open source event-based database called Sky (http://skydb.io http://skydb.io). Similar data model but it's aimed at a different use case than Datomic -- it's primarily used for analytics. The event data model is nice for avoiding conflicts but it's an up road battle to try and replace traditional relational databases. I find that storing historical events is primarily useful for either analytics or auditing. It makes more sense to move that off your primary data store if possible. I do find that making databases smarter about the data they hold can really provide some great optimizations. Sky can crunch through about 60MM events per second on a single core because it is built for event data models. I'm sure Datomic can probably get some good optimizations too. I'll be curious to see some benchmarks. Here's a demo app of Sky crunching all the events in the GitHub Archive (starts around 3m:20s): http://skydb.io/blog/github-archive-visualizer.html http://skydb.io/blog/github-archive-visualizer.html I'm working on multi-core distribution right now and I'll be doing multi-node after that. Check out the web site or GitHub (https://github.com/skydb/sky https://github.com/skydb/sky) for more information. It's MIT licensed so feel free to hack away.
- fizx 14y ago
  You really need a "how it works" section. Few people are willing to read the source code to see if you have interesting or useful ideas.
  - benbjohnson 14y ago
    That's a good point. I'm cleaning up and stabilizing the code right now and part of my v0.2.0 release is going to be documentation. I did a presentation recently about some of the ideas behind the database: https://speakerdeck.com/benbjohnson/behavioral-databases https://speakerdeck.com/benbjohnson/behavioral-databases The intro on the GitHub Archive Visualizer video explains some more of it too: https://vimeo.com/51629936 https://vimeo.com/51629936 I'll add that stuff to the README though.
chillax 14y ago
In related news Riak was in fact added as supported storage today: http://blog.datomic.com/2012/11/riak-and-couchbase-support.html http://blog.datomic.com/2012/11/riak-and-couchbase-support.h...

mvalle 14y ago

Sorry for going off-topic, but is there any explanation why InfoQ does not have an entry on Wikipedia? Is it just because no one has bothered to do it yet, or is there some other reason why this quite big, and frequently cited, site is not on Wikipedia?

krosaen 14y ago

Related: presentation on how datomic was written in clojure http://www.infoq.com/presentations/Datomic http://www.infoq.com/presentations/Datomic