4 ms·

Last week in the official cognitect training they said that the transactor can handle 10k writes/sec with appropriate hardware, which is enough to handle all th

Last week in the official cognitect training they said that the transactor can handle 10k writes/sec with appropriate hardware, which is enough to handle all the credit card transactions in the world. [This next part is me, not them] So despite being theoretically write bottlenecked, it still does better than postgres, because the transactor does less work (though they're in the same ballpark). What do you think? I hope I'm quoting them right, from memory.

coolsunglasses 13y ago
I could construct scenarios where identical hardware for PostgreSQL-alone and Datomic+PostgreSQL were alternately beating each other in transactions per second. The main limitations are that if you exceed the datoms the b-tree indexes can handle, you'll need to partition-walk or shard, and if you exceed the transactions/second a single fully vertically scaled Datomic transactor can handle, you'll need to evaluate sharding strategies similar to what PostgreSQL deployments do. I don't think a Datomic transactor in-and-of-itself is more efficient in $/performance for transactions per second than the same exact machine just running PostgreSQL. That's supposition, I don't have data conclusively proving that even though we directly compared some workloads on a realistic PostgreSQL schema vs. Datomic using PostgreSQL as the backend. The design does make Datomic being more efficient hypothetically possible, I just don't think it's yet the case, I could easily be proven wrong though. I don't think any difference between the two is meaningful other than the fact that if you created separate sharded transactor instances that would be with the caveat that you can no longer get a whole-database atomic view of what's going on. Which is the same thing that happens when you have multiple sharded writers in PostgreSQL, it's just that it's explicitly part of the point of Datomic, but if you only need an atomic view of shardable graphs of relationships of the historical data instead of the whole database, then the sky is hypothetically the limit. It'd still make me a little uncomfortable, but it's better than implementing your own event sourcing on Cassandra, haha. It's worth keeping in mind that if you just want to throw transactions over the wall from your client, you can do that with transact-async. HornetQ has no problem just holding onto the txes as they get worked through ;) The main thing I would keep in mind is that while you can get away with some modest OLAP'ish stuff, PostgreSQL is a lot better at OLAP (presently) than Datomic due to the way peers work. There's not a nice way currently to shard reads, even though that's the only kind of sharding that's really possible at present. You could dispatch range queries against peers that become responsible for their sections of the sharded ring, letting the LRU caching do the "population" per shard for you (with minimal penalties for exceeding the memory capacity for each shard). I could see getting pretty far with that alone. But again, Datomic is kinda like Lisp right now in the sense that it's a solid building material but there's no Staples easy-button Magic like, say, Riak at present. To do something like Redshift with Datomic would require query sharding (can be client-side library) against a buncha warmed up peers. You wouldn't want to just churn the data through a single peer just "doing" all the work. I'm perfectly happy to use Datomic as my default OLTP store and lazily replicate the data into something designed for OLAP workloads. Sorry for babbling, I just got back from a type theory meetup that I got a couple of fellow Clojurians to tag along with me for and we ended up having some insanely fun conversations with other FPers. I'm pumped tonight :) Apologies if I babbled so much that I didn't really answer your question, I hope I encompassed the general subject of write-throughput. tl;dr s/PostgreSQL/Datomic/g for OLTP workloads, despite having been a PGSQL nut for years. Possible biases: I hate MangoDB because I've actually used it in non-trivial production deployments. I really liked PGSQL for many years despite using it in non-trivial production deployments. I'm sort of a maintainer of Korma (SQL library for Korma). I'm an unapologetic Clojure and Haskell fan. I technically worked on a library for RethinkDB. I happen to think it's a nice document store.

coolsunglasses 13y ago

I could construct scenarios where identical hardware for PostgreSQL-alone and Datomic+PostgreSQL were alternately beating each other in transactions per second. The main limitations are that if you exceed the datoms the b-tree indexes can handle, you'll need to partition-walk or shard, and if you exceed the transactions/second a single fully vertically scaled Datomic transactor can handle, you'll need to evaluate sharding strategies similar to what PostgreSQL deployments do. I don't think a Datomic transactor in-and-of-itself is more efficient in $/performance for transactions per second than the same exact machine just running PostgreSQL. That's supposition, I don't have data conclusively proving that even though we directly compared some workloads on a realistic PostgreSQL schema vs. Datomic using PostgreSQL as the backend. The design does make Datomic being more efficient hypothetically possible, I just don't think it's yet the case, I could easily be proven wrong though. I don't think any difference between the two is meaningful other than the fact that if you created separate sharded transactor instances that would be with the caveat that you can no longer get a whole-database atomic view of what's going on. Which is the same thing that happens when you have multiple sharded writers in PostgreSQL, it's just that it's explicitly part of the point of Datomic, but if you only need an atomic view of shardable graphs of relationships of the historical data instead of the whole database, then the sky is hypothetically the limit. It'd still make me a little uncomfortable, but it's better than implementing your own event sourcing on Cassandra, haha. It's worth keeping in mind that if you just want to throw transactions over the wall from your client, you can do that with transact-async. HornetQ has no problem just holding onto the txes as they get worked through ;) The main thing I would keep in mind is that while you can get away with some modest OLAP'ish stuff, PostgreSQL is a lot better at OLAP (presently) than Datomic due to the way peers work. There's not a nice way currently to shard reads, even though that's the only kind of sharding that's really possible at present. You could dispatch range queries against peers that become responsible for their sections of the sharded ring, letting the LRU caching do the "population" per shard for you (with minimal penalties for exceeding the memory capacity for each shard). I could see getting pretty far with that alone. But again, Datomic is kinda like Lisp right now in the sense that it's a solid building material but there's no Staples easy-button Magic like, say, Riak at present. To do something like Redshift with Datomic would require query sharding (can be client-side library) against a buncha warmed up peers. You wouldn't want to just churn the data through a single peer just "doing" all the work. I'm perfectly happy to use Datomic as my default OLTP store and lazily replicate the data into something designed for OLAP workloads. Sorry for babbling, I just got back from a type theory meetup that I got a couple of fellow Clojurians to tag along with me for and we ended up having some insanely fun conversations with other FPers. I'm pumped tonight :) Apologies if I babbled so much that I didn't really answer your question, I hope I encompassed the general subject of write-throughput. tl;dr s/PostgreSQL/Datomic/g for OLTP workloads, despite having been a PGSQL nut for years. Possible biases: I hate MangoDB because I've actually used it in non-trivial production deployments. I really liked PGSQL for many years despite using it in non-trivial production deployments. I'm sort of a maintainer of Korma (SQL library for Korma). I'm an unapologetic Clojure and Haskell fan. I technically worked on a library for RethinkDB. I happen to think it's a nice document store.