4 ms·

Fixing Bad Data in Datomic

103 points by lauriswtf 12y ago

sargun 12y ago
I really like the fact that facts are immutable in the Datomic model. It actually makes it easier to reason about eventually consistent, because you can rely on causal consistency between facts. Eventually consistent systems can give you durability, isolation, and atomicity, but it's typically hard to reason about, because the interface that's given to the user is some sort of multi-value, vector-clock tracked thing, and it's hard to reason about. Datomic can manage a commit protocol under the hood, and deal with causality tracking without requiring the developer to "get dirty." It's query language is declarative, and doesn't provide the same implicit guarantees that SQL queries need.
- mdavidn 12y ago
  Datomic is not an eventually consistent system. All writes pass through a single "transactor," which serializes transactions and provides the same ACID guarantees as a traditional SQL database.
  - sargun 12y ago
    But, the data storage can be in a AP system, allowing better stories around data availability.
    - dantiberian 12y ago
      The Datomic docs at http://docs.datomic.com/acid.html http://docs.datomic.com/acid.html describe it's ACID properties quite well. > Another way to understand this is to consider the failure mode introduced by an eventually consistent node that is not up-to-date yet. Datomic will always see a correct log pointer, which was placed via conditional put. If some of the tree nodes are not yet visible underneath that pointer, Datomic is consistent but partially unavailable, and will become fully available when eventually happens.
solutionyogi 12y ago
That was a fantastic read. I can't count the number of hours I have wasted trying to figure out a bug (or trace a problem) in the system because someone decided to update the data silently. I think this is the kind of database which should be standard in finance industry. I have been working for financial companies in NYC for last 8 years and I have yet to see anyone use this product. Does anyone on HN have used this on a live system?
- chimeracoder 12y ago
  I heard Rich Hickey speak about a year ago about Datomic. Those are exactly the sorts of clients that Datomic aims to sell to (and their pricing model reflects that). Though Datomic is still relatively new, so knowing how quickly (read: slowly) financial institutions update their infrastructure, I'd be (pleasantly) surprised if it were widely deployed in a single financial institution yet. I really liked Datomic from what I could see - at the time we started building our stack at my startup it didn't mature enough (library support, etc.) to be worth it, but it could make our lives a lot easier.
- danneu 12y ago
  I ported my 7-year-old vBulletin community to a forum I wrote from scratch with Clojure + Datomic earlier this year. Datomic was a nice choice since it makes it trivial to expose a post's edit history. It also makes it easy to query for all the actions a moderator is responsible for. And if a moderator goes rogue and deletes everything, it's easy to revert.
  - solutionyogi 12y ago
    Thanks for sharing your experience. How much effort was it? Did you run in to bugs? How is the performance for a public facing website? How hard was to adopt to this new mindset about immutable database? Ever since I started using Git, I have been wondering as to why we don't have a database product which works like Git. When I read through this fantastic page on their rationale (http://www.datomic.com/rationale.html http://www.datomic.com/rationale.html), I was furiously nodding my head in agreement. Their approach makes so much sense in this day and age where hardware is cheap.
ds_ 12y ago
Are there any promising open source alternatives with similar philosophies to datomic?

103 points by lauriswtf 12y ago

I really like the fact that facts are immutable in the Datomic model. It actually makes it easier to reason about eventually consistent, because you can rely on causal consistency between facts. Eventually consistent systems can give you durability, isolation, and atomicity, but it's typically hard to reason about, because the interface that's given to the user is some sort of multi-value, vector-clock tracked thing, and it's hard to reason about. Datomic can manage a commit protocol under the hood, and deal with causality tracking without requiring the developer to "get dirty." It's query language is declarative, and doesn't provide the same implicit guarantees that SQL queries need.

mdavidn 12y ago
Datomic is not an eventually consistent system. All writes pass through a single "transactor," which serializes transactions and provides the same ACID guarantees as a traditional SQL database.
- sargun 12y ago
  But, the data storage can be in a AP system, allowing better stories around data availability.
  - dantiberian 12y ago
    The Datomic docs at http://docs.datomic.com/acid.html http://docs.datomic.com/acid.html describe it's ACID properties quite well. > Another way to understand this is to consider the failure mode introduced by an eventually consistent node that is not up-to-date yet. Datomic will always see a correct log pointer, which was placed via conditional put. If some of the tree nodes are not yet visible underneath that pointer, Datomic is consistent but partially unavailable, and will become fully available when eventually happens.

solutionyogi 12y ago

That was a fantastic read. I can't count the number of hours I have wasted trying to figure out a bug (or trace a problem) in the system because someone decided to update the data silently. I think this is the kind of database which should be standard in finance industry. I have been working for financial companies in NYC for last 8 years and I have yet to see anyone use this product. Does anyone on HN have used this on a live system?

chimeracoder 12y ago
I heard Rich Hickey speak about a year ago about Datomic. Those are exactly the sorts of clients that Datomic aims to sell to (and their pricing model reflects that). Though Datomic is still relatively new, so knowing how quickly (read: slowly) financial institutions update their infrastructure, I'd be (pleasantly) surprised if it were widely deployed in a single financial institution yet. I really liked Datomic from what I could see - at the time we started building our stack at my startup it didn't mature enough (library support, etc.) to be worth it, but it could make our lives a lot easier.
danneu 12y ago
I ported my 7-year-old vBulletin community to a forum I wrote from scratch with Clojure + Datomic earlier this year. Datomic was a nice choice since it makes it trivial to expose a post's edit history. It also makes it easy to query for all the actions a moderator is responsible for. And if a moderator goes rogue and deletes everything, it's easy to revert.
- solutionyogi 12y ago
  Thanks for sharing your experience. How much effort was it? Did you run in to bugs? How is the performance for a public facing website? How hard was to adopt to this new mindset about immutable database? Ever since I started using Git, I have been wondering as to why we don't have a database product which works like Git. When I read through this fantastic page on their rationale (http://www.datomic.com/rationale.html http://www.datomic.com/rationale.html), I was furiously nodding my head in agreement. Their approach makes so much sense in this day and age where hardware is cheap.

ds_ 12y ago

Are there any promising open source alternatives with similar philosophies to datomic?