8 ms·

Datomic by Rich Hickey [video]

94 points by sethev 13y ago

chimeracoder 13y ago
I haven't used Datomic at all myself, though I was at this event and I found it very appealing. The issue of having an easy audit trail is absolutely critical for many businesses, and is ill-served by most existing databases. This is one thing I liked about CouchDB - CouchDB does not enforce this strictly, the way Datomic does, but the model encourages "append to edit" rather than strict modification. I haven't had the time yet, but I'm excited to check out Datomic specifically for this reason.
juliangamble 13y ago
From watching the screencasts that Hickey produces for Clojure - he talks about working on a 'Real Time Broadcasting System' (National Election data streaming). It sounds like he spent lots of time thinking about how to guarantee what a real-time system would do when reporting on snapshots data as it was streaming through. The ideas in Clojure and Datomic seem to be the result of that thinking process. (ie in Datomic you have the ability to go back and see what the state of the system was at that point in time. In Datomic the agent model gives you the ability to see the state of a value in the middle of an STM transaction.)
coolsunglasses 13y ago
Happily using Datomic at work, released a Datomic migration library based on that work recently. (https://github.com/bitemyapp/brambling/ https://github.com/bitemyapp/brambling/) If you have any questions about pros/cons, what it's like to use, operational constraints/tradeoffs, please ask.
- iampims 13y ago
  I’d be curious to hear how Datomic scales, ops wise and application wise. Is adding a “peer” all there is to do to handle more reads?
  - coolsunglasses 13y ago
    The more specific you are about your concerns, the better. >Is adding a “peer” all there is to do to handle more reads? Yes, modulo storage backend throughput capacity. It uses storage backends in a fairly dumb and efficient manner so in all likelihood peers will be the bottleneck. >ops wise PostgreSQL and DynamoDB are pretty popular, the ops scenario for the latter is pretty obvious. You can just scale the provisioned IOPS according to your storage backend throughput. Pretty sure DynamoDB can exceed anything you'll ever need in that regard. A sufficiently beefy PostgreSQL machine (SSD, etc) can go ham too. The main shift in mentality with Datomic ops and application-wise is that you're prioritizing vertical scaling before horizontal scaling. I actually think that's the smarter thing to do from a cost and complexity stand-point anyway. I've worked in environments where horizontal scaling was deployed prematurely and we were managing 100 machines where ~3-5 would've sufficed. Part of this enhanced emphasis on scaling is that you're incentivized to make the most of each peer. If you have peers that are making "lightweight" use of (Datomic/your data) but you need many of them running (for compute?) then call into a REST peer or custom REST interface to read/write the data. You implement sprocs through database functions which are just Clojure code, super nice. Main scaling choke-point is just the transactor itself. >Application wise The two X-factors for Datomic in my experience are datalog and the deeply "historical" nature of the database and querying. Being able to arbitrarily time-slice queries with complicated relationships without having to change them in order to accommodate the "time-slicing" is amazing. There are separate interfaces for querying and transacting, which is nice. I would advise avoiding doing more migrations than necessary since that means you'll have to use my library Brambling though :) The Datomic guys are working on implementing trivial schema changes (like adding indexes), but for the more complicated migrations, you'll have to use Brambling or something comparable. Any other queries?
    - iampims 13y ago
      Thanks for the detailed reply. One more question: What's your failure recovery strategy when your writer fails?
      - brandonbloom 13y ago
- TheHydroImpulse 13y ago
  In his "Functional Database" talk he shows many examples of Datomic's snapshot feature. He keeps saying how the changes hasn't been committed yet, and is only available to the local client. When you take these snapshots, are you still touching the database? Or is the data held in memory in the application? It seems like the former makes a lot more sense, but he makes it seem like he's only working with a cloned, local copy.
  - dustingetz 13y ago
    Datomic is like git. You are working with a heavily lazy cloned copy, where only the parts that matter are cached locally. This "perfect" caching is enabled by an append-only data model, like git; queries don't touch network, like git. This has huge ramifications across the board, both obvious (omg free queries, horizontal read scalability) and subtle (no object relational impedance mismatch).
    - coolsunglasses 13y ago
      Free queries modulo the degree to which all your queries touch the same data and you can store the entities you touch in memory. You could hypothetically thrash the LRU by working with a large dataset and forcing the LRU to cyclically evict the data, but even the pathological case is at worst identical to how SQL databases work.
nabla9 13y ago
I think this might be good time to mention HiBase and Shades projects and related research for data structures and functional programming environments. http://www.cs.hut.fi/~siva/hibase/ http://www.cs.hut.fi/~siva/hibase/ HiBase was project funded by Nokia. The Goal was to develop a persistent functional programming environment with for telecom environments. The reason why it was functional was the observation that in functional programming environment you can model data as graph where references point only backwards in time. In other words, if you just constantly write your data into the database and have smart GC, you can recover very fast from crashes. Just like telecom environment likes it. The project was canceled, some say for reasons to do with internal company politics, but it showed huge promise. It was already 10X faster than competing commercial databases that were tweaked by their developers to perform well in testing.
Estragon 13y ago
Anything new here for people who've seen him talk on Datomic before?
- brandonbloom 13y ago
  I think this is the same talk he gave at another NYC meetup a few months ago. The beginning of the talk is an interesting take on thinking about "processes" and "machines", in true Rich Hickey style. The later half of the talk involves some REPL exploration, which gives a good flavor of actually using the thing.
  - Estragon 13y ago
    Thanks, Brandon.

94 points by sethev 13y ago

I haven't used Datomic at all myself, though I was at this event and I found it very appealing. The issue of having an easy audit trail is absolutely critical for many businesses, and is ill-served by most existing databases. This is one thing I liked about CouchDB - CouchDB does not enforce this strictly, the way Datomic does, but the model encourages "append to edit" rather than strict modification. I haven't had the time yet, but I'm excited to check out Datomic specifically for this reason.

juliangamble 13y ago

From watching the screencasts that Hickey produces for Clojure - he talks about working on a 'Real Time Broadcasting System' (National Election data streaming). It sounds like he spent lots of time thinking about how to guarantee what a real-time system would do when reporting on snapshots data as it was streaming through. The ideas in Clojure and Datomic seem to be the result of that thinking process. (ie in Datomic you have the ability to go back and see what the state of the system was at that point in time. In Datomic the agent model gives you the ability to see the state of a value in the middle of an STM transaction.)

coolsunglasses 13y ago

Happily using Datomic at work, released a Datomic migration library based on that work recently. (https://github.com/bitemyapp/brambling/ https://github.com/bitemyapp/brambling/) If you have any questions about pros/cons, what it's like to use, operational constraints/tradeoffs, please ask.

iampims 13y ago
I’d be curious to hear how Datomic scales, ops wise and application wise. Is adding a “peer” all there is to do to handle more reads?
- coolsunglasses 13y ago
  The more specific you are about your concerns, the better. >Is adding a “peer” all there is to do to handle more reads? Yes, modulo storage backend throughput capacity. It uses storage backends in a fairly dumb and efficient manner so in all likelihood peers will be the bottleneck. >ops wise PostgreSQL and DynamoDB are pretty popular, the ops scenario for the latter is pretty obvious. You can just scale the provisioned IOPS according to your storage backend throughput. Pretty sure DynamoDB can exceed anything you'll ever need in that regard. A sufficiently beefy PostgreSQL machine (SSD, etc) can go ham too. The main shift in mentality with Datomic ops and application-wise is that you're prioritizing vertical scaling before horizontal scaling. I actually think that's the smarter thing to do from a cost and complexity stand-point anyway. I've worked in environments where horizontal scaling was deployed prematurely and we were managing 100 machines where ~3-5 would've sufficed. Part of this enhanced emphasis on scaling is that you're incentivized to make the most of each peer. If you have peers that are making "lightweight" use of (Datomic/your data) but you need many of them running (for compute?) then call into a REST peer or custom REST interface to read/write the data. You implement sprocs through database functions which are just Clojure code, super nice. Main scaling choke-point is just the transactor itself. >Application wise The two X-factors for Datomic in my experience are datalog and the deeply "historical" nature of the database and querying. Being able to arbitrarily time-slice queries with complicated relationships without having to change them in order to accommodate the "time-slicing" is amazing. There are separate interfaces for querying and transacting, which is nice. I would advise avoiding doing more migrations than necessary since that means you'll have to use my library Brambling though :) The Datomic guys are working on implementing trivial schema changes (like adding indexes), but for the more complicated migrations, you'll have to use Brambling or something comparable. Any other queries?
  - iampims 13y ago
    Thanks for the detailed reply. One more question: What's your failure recovery strategy when your writer fails?
    - brandonbloom 13y ago
TheHydroImpulse 13y ago
In his "Functional Database" talk he shows many examples of Datomic's snapshot feature. He keeps saying how the changes hasn't been committed yet, and is only available to the local client. When you take these snapshots, are you still touching the database? Or is the data held in memory in the application? It seems like the former makes a lot more sense, but he makes it seem like he's only working with a cloned, local copy.
- dustingetz 13y ago
  Datomic is like git. You are working with a heavily lazy cloned copy, where only the parts that matter are cached locally. This "perfect" caching is enabled by an append-only data model, like git; queries don't touch network, like git. This has huge ramifications across the board, both obvious (omg free queries, horizontal read scalability) and subtle (no object relational impedance mismatch).
  - coolsunglasses 13y ago
    Free queries modulo the degree to which all your queries touch the same data and you can store the entities you touch in memory. You could hypothetically thrash the LRU by working with a large dataset and forcing the LRU to cyclically evict the data, but even the pathological case is at worst identical to how SQL databases work.

nabla9 13y ago

I think this might be good time to mention HiBase and Shades projects and related research for data structures and functional programming environments. http://www.cs.hut.fi/~siva/hibase/ http://www.cs.hut.fi/~siva/hibase/ HiBase was project funded by Nokia. The Goal was to develop a persistent functional programming environment with for telecom environments. The reason why it was functional was the observation that in functional programming environment you can model data as graph where references point only backwards in time. In other words, if you just constantly write your data into the database and have smart GC, you can recover very fast from crashes. Just like telecom environment likes it. The project was canceled, some say for reasons to do with internal company politics, but it showed huge promise. It was already 10X faster than competing commercial databases that were tweaked by their developers to perform well in testing.

Estragon 13y ago

Anything new here for people who've seen him talk on Datomic before?

brandonbloom 13y ago
I think this is the same talk he gave at another NYC meetup a few months ago. The beginning of the talk is an interesting take on thinking about "processes" and "machines", in true Rich Hickey style. The later half of the talk involves some REPL exploration, which gives a good flavor of actually using the thing.
- Estragon 13y ago
  Thanks, Brandon.