5 ms·

Having built a variety of products with Datomic and performance requirements I can only disagree. Especially queries are insanely fast compared to other databas

Having built a variety of products with Datomic and performance requirements I can only disagree. Especially queries are insanely fast compared to other databases. I don't know what your data team was doing, but they must have done it wrong. My advice is to develop without a data team from the start ;)

nas 9y ago
Datomic is interesting. It is somewhat similar to object databases like ZODB. I use an object database called Durus (spiritual derivative of ZODB). Both ZODB and Datomic make use of aggressive caching in the application. That is a huge performance boost for applications that do a lot of reading and little writing. ZODB essentially puts the indexes in the application, rather than having the DB layer take care of indexes. Again, I understand this is similar to how Datomic works. Developing an application on top of ZODB vs on top of an SQL database leads to a very different data model design. You can't just swap one database layer out for the other and expect things to work well. I'm not surprised if a team who is used to developing on top of SQL-like DBs did not have good luck when moving to Datomic. It is just a very different model.
qaq 9y ago
would you give an example?
- lgrapenthin 9y ago
  The main advantage is that you can do time-consistent queries against any past database state across the entire database without any read locks. This frees resources and eliminates the necessity of timestamp tables. Then there is Datomics caching model in combination with its local data based query engine. The vast majority of critical queries only read from memory. Data that is fetched doesn't block other consumers.
  - qaq 9y ago
    What RDBMS are you comparing to pretty much everything is MVCC so writers don't block readers and readers do not block writers mature RDBMS also are fairly highly optimized written in C with critical paths hand written in assembly so again would be interesting to learn for what workload Datomic is significantly faster. The biggest performance penalty in a modern RDBMS is GC of snapshots so if you need a design that is append only it will make modern RDBMS more performant.
    - didibus 9y ago
      I think a big difference is that Datomic can scale out reads in a distributed way. While also keeping writes consistent and transactional. So its specifically designed for such case where you need consistent transactional writes, while needing high scale reads.
      - qaq 9y ago
        PostgreSQL can do few million reads per second on a high end x86 box. You can scale our reads easily by running multiple slaves.
    - lgrapenthin 9y ago
      I don't care so much about optimized tight loops written in assembly, rather about the ability to scale nowadays. Datomic uses databases like you describe e.g. Postgres as its storage. So it would be foolish to say that it beats their performance on a bare bones level. Instead, Datomics architecture and information model make it much easier and significantly reduce the overhead to design and implement applications that provide insane performance. I won't argue that you can hand rewrite every Datomic application in its underlying storage database and get more performance out of it if you do your caching and coordination right. It will take you much longer though (I'd guess a tenfold at least), likely have some very difficult to find bugs, and the result won't be as easy to extend. With Datomic, I get memory speed performance out of the box for the heavy hitters and so much more that it take would take some very uncommon requirements for me to choose something else nowadays.
      - 9y ago
carterehsmith 9y ago
>> Especially queries are insanely fast compared to other databases. Nice! Insanely fast? Awesome. What kind of queries compared to what databases?

nas 9y ago

Datomic is interesting. It is somewhat similar to object databases like ZODB. I use an object database called Durus (spiritual derivative of ZODB). Both ZODB and Datomic make use of aggressive caching in the application. That is a huge performance boost for applications that do a lot of reading and little writing. ZODB essentially puts the indexes in the application, rather than having the DB layer take care of indexes. Again, I understand this is similar to how Datomic works. Developing an application on top of ZODB vs on top of an SQL database leads to a very different data model design. You can't just swap one database layer out for the other and expect things to work well. I'm not surprised if a team who is used to developing on top of SQL-like DBs did not have good luck when moving to Datomic. It is just a very different model.

qaq 9y ago

would you give an example?

lgrapenthin 9y ago
The main advantage is that you can do time-consistent queries against any past database state across the entire database without any read locks. This frees resources and eliminates the necessity of timestamp tables. Then there is Datomics caching model in combination with its local data based query engine. The vast majority of critical queries only read from memory. Data that is fetched doesn't block other consumers.
- qaq 9y ago
  What RDBMS are you comparing to pretty much everything is MVCC so writers don't block readers and readers do not block writers mature RDBMS also are fairly highly optimized written in C with critical paths hand written in assembly so again would be interesting to learn for what workload Datomic is significantly faster. The biggest performance penalty in a modern RDBMS is GC of snapshots so if you need a design that is append only it will make modern RDBMS more performant.
  - didibus 9y ago
    I think a big difference is that Datomic can scale out reads in a distributed way. While also keeping writes consistent and transactional. So its specifically designed for such case where you need consistent transactional writes, while needing high scale reads.
    - qaq 9y ago
      PostgreSQL can do few million reads per second on a high end x86 box. You can scale our reads easily by running multiple slaves.
  - lgrapenthin 9y ago
    I don't care so much about optimized tight loops written in assembly, rather about the ability to scale nowadays. Datomic uses databases like you describe e.g. Postgres as its storage. So it would be foolish to say that it beats their performance on a bare bones level. Instead, Datomics architecture and information model make it much easier and significantly reduce the overhead to design and implement applications that provide insane performance. I won't argue that you can hand rewrite every Datomic application in its underlying storage database and get more performance out of it if you do your caching and coordination right. It will take you much longer though (I'd guess a tenfold at least), likely have some very difficult to find bugs, and the result won't be as easy to extend. With Datomic, I get memory speed performance out of the box for the heavy hitters and so much more that it take would take some very uncommon requirements for me to choose something else nowadays.
    - 9y ago

carterehsmith 9y ago

>> Especially queries are insanely fast compared to other databases. Nice! Insanely fast? Awesome. What kind of queries compared to what databases?