4 ms·

I am not sure how this would work in a production database but the problem of concurrent reads and writes are pretty well solved in Clojure. https://clojure.org

I am not sure how this would work in a production database but the problem of concurrent reads and writes are pretty well solved in Clojure. https://clojure.org/reference/refs https://clojure.org/reference/refs I was also wondering if the Datomic approach is better.

stingraycharles 8y ago
Clojure's refs are actually STM based on MVCC, as explained in the link you mentioned. The problem with it is that it's really, really slow. It works elegantly and if performance is not too big of an issue, it's an appropriate solution. There's a reason why Datomic is generally known as being quite slow, and Cognitect being really insistent on not allowing anyone to publish any benchmarks, ever. (As a sidenote, I'm a long term Clojure developer and an absolute fan of Datomic's data model, but the context of the original article is all about speed)
- StreamBright 8y ago
  Well just saying something is slow is a little hand wavy. I understand that writing a database in Clojure/Java is not great (hello Cassandra, Hadoop, Hbase, ....) but in my Clojure code I found STM rather useful for sharing the same reference and it was always enough fast for the use cases I was running into. I guess Datomic might be slow, I was more referring to the idea of using a transactor to manage the concurrent writes as opposed to the model the was proposed in the video. Splitting up your database to different parts makes sense to me.
Scarbutt 8y ago
Datomic is not a database you want if you have any performance requirements. Tuning it for production is a nightmare, only cognitect can do it, so if you use it for anything serious, make sure to buy a support contract. They partly solve the 'no one can tune this black box' with their cloud offering by doing it for you, but cloud is a different product and dependent on AWS, it doesn't have an on-premise version.
- StreamBright 8y ago
  I see. Just like Hadoop, Cassandra, Hbase, MySQL, Couchbase and many other systems. You might pull out the opensource card but there are companies live out of supporting these great opensource systems because it is not trivial to install, configure and tune them. I am not sure how this new system will be any different.
- dustingetz 8y ago
  If you look at what Reddit [1] and Facebook [2] did in their data layers to scale past what the relational model can offer, they start to look pretty darn close to the Datomic architecture. Linearized. Time aware. Immutable. Reads from cache. Horizontal scaling. Query/writer/storage separation. And obviously they are cloud architectures, built and operated by teams of distributed systems engineers capable of tuning performance of such a system. [1] https://news.ycombinator.com/item?id=15726376 https://news.ycombinator.com/item?id=15726376 [2] http://www.dustingetz.com/:datomic-facebook-tao/ http://www.dustingetz.com/:datomic-facebook-tao/ Neither of these architectures resemble the relational model anymore. They resemble a half-baked bug-ridden implementation of half of Datomic. TLDR: Nobody likes their database and zero teams at scale run vanilla out-of-box database. This might shed light on what people mean when they are talking about tuning Datomic: http://www.dustingetz.com/:datomic-performance-gaare/ http://www.dustingetz.com/:datomic-performance-gaare/ I'm sure there is more but this is the only thing I have ever come across.
frankmcsherry 8y ago
If you want to check out more about high-throughput incremental view maintenance in a Datomic-like system, here's a decent read (caveat: I'm involved): https://www.nikolasgoebel.com/2018/09/13/incremental-datalog.html https://www.nikolasgoebel.com/2018/09/13/incremental-datalog... Also, a recent video from ClojureConj on the subject: https://www.youtube.com/watch?v=ZgqFlowyfTA https://www.youtube.com/watch?v=ZgqFlowyfTA

stingraycharles 8y ago

Clojure's refs are actually STM based on MVCC, as explained in the link you mentioned. The problem with it is that it's really, really slow. It works elegantly and if performance is not too big of an issue, it's an appropriate solution. There's a reason why Datomic is generally known as being quite slow, and Cognitect being really insistent on not allowing anyone to publish any benchmarks, ever. (As a sidenote, I'm a long term Clojure developer and an absolute fan of Datomic's data model, but the context of the original article is all about speed)

StreamBright 8y ago
Well just saying something is slow is a little hand wavy. I understand that writing a database in Clojure/Java is not great (hello Cassandra, Hadoop, Hbase, ....) but in my Clojure code I found STM rather useful for sharing the same reference and it was always enough fast for the use cases I was running into. I guess Datomic might be slow, I was more referring to the idea of using a transactor to manage the concurrent writes as opposed to the model the was proposed in the video. Splitting up your database to different parts makes sense to me.

Scarbutt 8y ago

Datomic is not a database you want if you have any performance requirements. Tuning it for production is a nightmare, only cognitect can do it, so if you use it for anything serious, make sure to buy a support contract. They partly solve the 'no one can tune this black box' with their cloud offering by doing it for you, but cloud is a different product and dependent on AWS, it doesn't have an on-premise version.

StreamBright 8y ago
I see. Just like Hadoop, Cassandra, Hbase, MySQL, Couchbase and many other systems. You might pull out the opensource card but there are companies live out of supporting these great opensource systems because it is not trivial to install, configure and tune them. I am not sure how this new system will be any different.
dustingetz 8y ago
If you look at what Reddit [1] and Facebook [2] did in their data layers to scale past what the relational model can offer, they start to look pretty darn close to the Datomic architecture. Linearized. Time aware. Immutable. Reads from cache. Horizontal scaling. Query/writer/storage separation. And obviously they are cloud architectures, built and operated by teams of distributed systems engineers capable of tuning performance of such a system. [1] https://news.ycombinator.com/item?id=15726376 https://news.ycombinator.com/item?id=15726376 [2] http://www.dustingetz.com/:datomic-facebook-tao/ http://www.dustingetz.com/:datomic-facebook-tao/ Neither of these architectures resemble the relational model anymore. They resemble a half-baked bug-ridden implementation of half of Datomic. TLDR: Nobody likes their database and zero teams at scale run vanilla out-of-box database. This might shed light on what people mean when they are talking about tuning Datomic: http://www.dustingetz.com/:datomic-performance-gaare/ http://www.dustingetz.com/:datomic-performance-gaare/ I'm sure there is more but this is the only thing I have ever come across.

frankmcsherry 8y ago

If you want to check out more about high-throughput incremental view maintenance in a Datomic-like system, here's a decent read (caveat: I'm involved): https://www.nikolasgoebel.com/2018/09/13/incremental-datalog.html https://www.nikolasgoebel.com/2018/09/13/incremental-datalog... Also, a recent video from ClojureConj on the subject: https://www.youtube.com/watch?v=ZgqFlowyfTA https://www.youtube.com/watch?v=ZgqFlowyfTA