4 ms·

For those that don't know, Datomic is a hosted database, similar to how Clojure is hosted. Datomic can be hosted on MySQL or PostgreSQL (and maybe others?). It

For those that don't know, Datomic is a hosted database, similar to how Clojure is hosted. Datomic can be hosted on MySQL or PostgreSQL (and maybe others?). It's basically a 2 column table, so yes, Datomic inherits transaction safety. According to people I know that use Datomic, an early lesson is indexing, which apparently is often done way later than it should. Datomic also inherits the speed of indexing a huge table. A counter is that many things may be covered in majic, but all on top of a fairly leaky abstraction.

augustl 8y ago
An interesting point to add is that Datomic can also write to Riak. But since riak lacks an atomic CAS, zookeeper is used alongside Riak. What do you refer to when you say that Datomic inherits the speed of indexing a huge table? The query engine works by fetching blocks of data from storage and placing the blocks in a lazy immutable tree structure that are the indices of Datomic. They are also covering (the indices is where the actual data is stored). So it's not like the query engine runs queries on the huge table stored in mysql or postgres, it's used more like a key/value store. Case in point: you can put memcached between the datomic client (peer) and any of the supported storage engines. So Datomic piggybacks on very little other than the actual capabiliy to safely store data.
- dizzystar 8y ago
  I don't know the full details, but I'm lead to understand that indexing a Datomic database too late is time-consuming and will end up causing considerable downtime. I'll admit that I'm guessing about how much of this depends on the database and Datomic itself, so I shouldn't be making a sweeping statement.
  - augustl 8y ago
    Ah, that makes sense. I've never added an index myself, but essentialy Datomic has to rewrite the main index tree. In other words, all of the stored data, since there is no data stored outside of the covering indies. So I wouldn't be surpised if that causes some down time if you have lots of data in your database.
  - ajss 8y ago
    I think "late indexing" maybe referring to an implementation detail. The indexes aren't updated after each transaction. When you query the db, the result is a merge between the index and the transactions that haven't been indexed yet. The index is updated in batches.
catnaroek 8y ago
It is not clear to me how you could enforce invariants that relate multiple database states. Allow me to give a silly toy example. Suppose you have to design a system where (0) Every bank account must admit a linear transaction history. (1) Given a causally related collection of transactions, all involved parties must agree on the relative order of these transactions. How exactly do you get to enforce invariants like these without transaction coordination?
dustingetz 8y ago
Datomic Cloud indexes automatically and persists to DynamoDB; indexing happens in a background process

augustl 8y ago

An interesting point to add is that Datomic can also write to Riak. But since riak lacks an atomic CAS, zookeeper is used alongside Riak. What do you refer to when you say that Datomic inherits the speed of indexing a huge table? The query engine works by fetching blocks of data from storage and placing the blocks in a lazy immutable tree structure that are the indices of Datomic. They are also covering (the indices is where the actual data is stored). So it's not like the query engine runs queries on the huge table stored in mysql or postgres, it's used more like a key/value store. Case in point: you can put memcached between the datomic client (peer) and any of the supported storage engines. So Datomic piggybacks on very little other than the actual capabiliy to safely store data.

dizzystar 8y ago
I don't know the full details, but I'm lead to understand that indexing a Datomic database too late is time-consuming and will end up causing considerable downtime. I'll admit that I'm guessing about how much of this depends on the database and Datomic itself, so I shouldn't be making a sweeping statement.
- augustl 8y ago
  Ah, that makes sense. I've never added an index myself, but essentialy Datomic has to rewrite the main index tree. In other words, all of the stored data, since there is no data stored outside of the covering indies. So I wouldn't be surpised if that causes some down time if you have lots of data in your database.
- ajss 8y ago
  I think "late indexing" maybe referring to an implementation detail. The indexes aren't updated after each transaction. When you query the db, the result is a merge between the index and the transactions that haven't been indexed yet. The index is updated in batches.

catnaroek 8y ago

It is not clear to me how you could enforce invariants that relate multiple database states. Allow me to give a silly toy example. Suppose you have to design a system where (0) Every bank account must admit a linear transaction history. (1) Given a causally related collection of transactions, all involved parties must agree on the relative order of these transactions. How exactly do you get to enforce invariants like these without transaction coordination?

dustingetz 8y ago

Datomic Cloud indexes automatically and persists to DynamoDB; indexing happens in a background process