9 ms·
Statistically (and from experience) I'm probably the weird one here, but I cannot fathom why Datomic isn't more popular. I get that postgres is a good default
by augustl 2y ago
Statistically (and from experience) I'm probably the weird one here, but I cannot fathom why Datomic isn't more popular.
I get that postgres is a good default in many cases, and I don't expect SQL to die tomorrow. But there are _so many_ apps (most/all backoffice apps I've worked on for example) with 10s or 100s of transactions per second at most, that would love to have the data available directly inside your business logic, and where both business logic and devops would improve by many orders of magnitude by having a full transaction log of all changes done to your data over time.
Is it _just_ because Datomic is different and people don't get it, and that preconceived notions makes you think Datomic is something it isn't?
Here's to the crazy ones!
- deleted 2y ago[deleted]
- Skinney 2y agoPart of the reason is that it's been a commercial database until recently. I also think it hasn't helped that it's closely associated with Clojure (for natural reasons), which might make Java developers turn away from it. It's a shame, though. Datomic is a marvel.
- intrepidpar 2y agoDatatomic is not open-source, is it? And I think the licensing model was a bit odd. Those definitely don't help.
- sswezey 2y agoCorrect. It's free, but not open-source.
- Arcanum-XIII 2y agoPricing killed it for my experimental project. Of course the complexity of having to set a cluster early on doesn’t help for small shops. Then there’s the complexity of learning datalog - love it, but when I see people already struggling with SQL, I’m not confidant about using it generally !
- augustl 2y agoIt's free now - but I suppose 10+ years of being both closed source and paid didn't do wonders for adoption. What part needs a cluster in your experience? The Datomic deployments I've been involved with have been running on a single server, with a single instance of backing storage etc. Datalog is interesting indeed... Back when I first started using Datomic in 2012 I had just fundamentally decided to use it, and it took probably a week before the query language "clicked", i.e. to be able to actually compose my own queries and not just copy paste my way to something that works.
- koito17 2y agoProfessional Clojure dev for some time. Here is what prevents me from using Datomic. - no Docker image; still distributed as a tarball. Although com.datomic/local exists, it only provides the Client API, so it's mostly suited towards mocking Datomic Cloud. - Datomic Cloud is designed around AWS; applications wanting to use Datomic Cloud must also design themselves around AWS - Datomic On-Prem does not scale the same way Datomic Cloud does (e.g. every database has an in-memory index; the transactor and *all peers* need to keep them all in-memory) - No query planner whatsoever. In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them. In Datomic, swapping two :where clauses can transform a 10ms query into a 10s query. In addition to the above four points, I strongly believe the following points prevent others from using Datomic. - Writing Datomic queries outside of Clojure (e.g. Java) requires writing everything in strings, which feels awful no matter what. - If you are not using a JVM-based language, then there is no choice but the REST API for interaction. The REST API is orders of magnitude slower than the official Clojure and Java clients. - Too many tooling exists around the pgwire protocol and various dialects of SQL. Datomic obviously does not fit into either of these categories of tooling. - Applications like DBeaver do not support Datomic at all. The best you can do is access the datomic_kvs table from a database using JDBC storage engine.
- refset 2y ago> In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them Rich's observation was that query optimizers often get things wrong in ways that are hard to predict or control, but he's not fundamentally opposed to their use. That said, building a decent optimizer is a huge undertaking and I think they took the right decision to not attempt to bundle that sort of complexity into the original vision for Datomic otherwise they might never have shipped. The state-of-the-art commercial engine for query optimization and execution algorithms in this 'triple' space is probably https://relational.ai/ https://relational.ai/
- marcinzm 2y agoAs I see it, Rich's stance is that of an expert in the database that doesn't need to deliver business features using the database. New users are not experts and even experienced users that work for companies have pressure to deliver features. You can get initial popularity by targeting these types of expert users working on more experimental products. However long term growth and popularity requires targeting the other 99.9% of users. I've seen one company adopt Datomic due to this type of user and then a couple years latter rip it out because as it grew it's developers were no longer of this type.
- emmanueloga_ 2y ago“Simple” comes from simplex (Latin for “not braided”), while “complex” means “braided together.” Datomic? Well, it feels like a whole lot of braiding going on. Datomic: complex made hard? Datomic requires a storage backend (e.g., Cassandra, DynamoDB, MySQL, etc.), transactors, peers, and a separate transaction log... also, it appears the few real users of Datomic are primarily running it in AWS. Trying to run it elsewhere likely introduces more complications and uncertainties. Good luck getting support, too—you might have to wait for Rich Hickey to finish his hammock nap (ba-dum-tss) While Datomic's immutability, time-travel queries, and Datalog query language are conceptually cool, they seem like niche features. My guess is that in 99% of the cases SQL can do the job just fine. Also, Rich Hickey may not have braids, but those curls come pretty close. Without deep pockets for high-RAM servers, many tend to steer clear of Java. If Datomic were packaged as a single binary with fewer dependencies (and no JVM requirement), it could attract more users. As it stands, it's a complex setup with limited benefits, primarily suited for specific use cases, I think (financial auditing / historical data analysis?). --- COLOPHON (from the Greek word κολοφών [kolophōn], meaning "summit" or "finishing touch") I don’t have real-world experience with Datomic; I played around with Datascript and ultimately steered away from Datomic after perceiving a low ROI given its complexity. I hope my fellow fans of Clojure and the venerable 'Simple Made Easy' talk can forgive this little attempt at humor at Rich Hickey's expense! :-p
- watt 2y ago"complex made hard" would be great. it seems you have missed the point of the talk: complex is always easy, complex is always where the gravitation pulls you towards.
- augustl 2y ago> Without deep pockets for high-RAM servers I just got a dedicated server on Hetzner to test out some things. It's $70/month with 64gb RAM and a CPU that builds a complex C++ thingie in 11 minutes where my laptop spends about an hour :D Scaling is of course not trivial, but the same set of backoffice apps I've worked with throughout the years that would be a good fit for Datomic, has a working set for the database much smaller than 64GB. > they seem like niche features That's the thing, though. Maybe it's because I've use Datomic a bunch. When I'm on projects that use a SQL db, a handful of problems are just fundamentally solved in Datomic, and none of the super knowledgeable devs that know SQL in and out are even aware that they are problems. Some examples: - What caused this column of this row to end up with this value? - Oh no, we were down 3am but when the first person investigated it at 7am (oh those backoffice SLAs...) everything works fine and nobody knows that the db state was at 3am - When we wrote value X, what other values did we also write at the same time? - We need to carefully write a transaction that reads some data, writes some data, and reads some more data, and hopefully lock correctly, and hopefully we understood the isolation level (that we probably didn't set anyway) correctly and... Which makes me think I must be the crazy one...
- benrutter 2y agoMy best guess is Datomic's coupling to the JVM makes it great for clojure developers (or java, scala etc) but not even considered by python or javascript shops, which make up the vast majority. If python and java has better interop, I think the story would be a little different maybe? My experience is that java is always a little stubborn as "child code" so libaries like pyspark often involve notoriously difficult set up.
- ndr 2y agoI love Clojure and would love to use Datomic. The fact that it is not open source pushes me away. The risk of being locked in is far too great for anything serious. There's no Valkey move if someone captures Datomic.
- Arjuna144 2y agoAbsolutely true! "No Valkey move", I like it! That should be come it's own meme! Being free but not open source makes no sense at all. It is just our little ego that wants to "keep" what we think of as "ours" to ourselves. Give all you have to all and progress will go so much faster. Others may learn from the implementation of Datomic and make something better. So your ego is hurt, but humanity wins!! Give all you have, and it will never be enough! Give anyway!
- dustingetz 2y agoRich said (in a random podcast, which was not widely circulated - possibly Cognicast) that their mission with Datomic Cloud was for it to be the easiest way to get started with clojure. He even used the word “Easy”! (unless i misremember, this needs to be checked). Anyway, Cloud had consumption based pricing and developers, companies, and even hobbyists are all accustomed to paying for managed cloud infrastructure. To me this seems like a viable strategy, and devs would have forgiven the previous licensing issues -- we just want to get paid to play with cool toys at work, you gotta work the money stuff out with the bosses. The problem with Cloud was that it didn’t deliver on easy. It was not the Heroku they envisioned, rather it was a scary AWS amalgamation that required deep AWS knowledge to debug their CloudFormation templates to get it to work. This is not even really their fault!, as CloudFormation is a dumpster fire, there are much better cloud orchestration tools now. And while AWS's first few products were rock solid (S3, EC2, DynamoDB), around this time is when AWS turned on the enterprise growth machine and pivoted into box checking and everything cloud began to turn to quicksand underneath their feet. On top of cloud quicksand, Datomic Cloud had technical scalability challenges which they brute force engineered through, motivating the architectural shift away from the easy synchronous/local entity api to the annoying async/remote client api, and then back again sorta with Ions -- but at the cost of, like, 3 years of product roadmap, during which time Typescript exploded in popularity, the serverless hype cycle began to break and SQL started making a comeback. Also, the Java API had poor adoption. selling predominantly into Clojure commercial users isn’t a great market so consequently: no VC, small team, no sales/marketing/devrel - at a time when the startup boom was gaining momentum, developer infra was getting funded for the first time so competitors were spending tons of money on beautiful docs, marketing, full time twitter accounts, etc. I still think there’s elements of an amazing product here, but the business window for a Clojure database has kinda closed - because Clojure itself has lost momentum. Datomic Cloud at its essence is a clojure hosting platform, the business is Clojure itself. I am very bullish on Clojure, I think it never found its killer app, and once found, once a money vector is opened, a bunch of cash (like, $100M†) will make all these problems go away and deliver on Clojure's mission to bring "simple made easy" to mass market application development, which is desperately needed. Maybe Hyperfiddle & Electric? † Paraphrasing a Materialize press release, “it takes $100M to build a production ready database” -- and for the record, Materialize isn't doing too well, nobody seems to know what it is for and the VCs replaced the CEO earlier this year. Full quote: "Why did we raise $100M? Put quite simply, we believe this is the order of magnitude of investment that it takes to build a production-ready database. Databases are notoriously hard to get right, and we do not intend to cut any corners." In the end, Cognitect chose to spend their lives doing something hard that matters, and I deeply respect that.
- pjmlp 2y agoI guess because those of us that are fine with Datomic, also don't have any issues going into IBM, Oracle or Microsoft RDMS portfolio of database products.
- TacticalCoder 2y ago> I get that postgres is a good default in many cases ... And Datomic can run on top of PostGres anyway right? So it's not as there wasn't a lot of the knowledge that wasn't reusable: not for the queries themselves but everything about managing the database. Or am I wrong here?
- augustl 2y agoYes - if I understand you correctly :) Datomic writes opaque blobs of index data to existing storage, such as Postgres or dynamodb or a handful of others. But you can't query that data meaningfully directly, it's just a bunch of compressed datomic-specific index data in there, no domain structure or query power etc.
- imhoguy 2y agoTabular data model, thus also relational, is in use since dawn of known civilization [0]. Anything what can't be described and literally touched as table causes discomfort and will do for centuries to go. This is why CSV is still the king in data transfer. Data outlives logic. Simple data structures survive complex data structures. And when you hardly tie data retrieval with complex logic and specific technology then one day you may not be able to simply retrieve it. ~~~ "ODE is a database system and environment based on the object paradigm. It offers one integrated data model for both database and general purpose manipulation. The database is defined, queried and manipulated in the database programming language O++ which is based on C++. O++ borrows and extends the object definition facility of C++, called the class. Classes support data encapsulation and multiple inheritance. We provide facilities for creating persistent and versioned objects, defining sets, and iterating over sets and clusters of persistent objects. We also provide facilities to associate constraints and triggers with objects. This paper presents the linguistic facilities provided in O++ and the data model it supports." 1989, [1] Do you see similarities? It even started similarly as a research project. And it is gone now, both logic and data, likely gone with last program using O++, and will be with Clojure. [0] https://www.datafix.com.au/BASHing/2020-08-12.html https://www.datafix.com.au/BASHing/2020-08-12.html [1] https://dl.acm.org/doi/abs/10.1145/66926.66930 https://dl.acm.org/doi/abs/10.1145/66926.66930
- panick21_ 2y agoAnd multiple table, connected make graphs. Almost nothing is a single table. If you want a single flat table in datomic, you can have it. In fact its literally just a big flat table conceptually. > And when you hardly tie data retrieval with complex logic and specific technology then one day you may not be able to simply retrieve it. You don't know how Datomic actually works do you?
- jwr 2y agoClojure developer here. I make a living from my SaaS, which is written in Clojure. Reasons why I don't use Datomic: * I've been burned in the past by a not-very-popular closed-source expensive database developed by a small company (specifically, Franz Inc's AllegroCache used with AllegroCL). * I don't actually want to preserve all history. I am worried about performance. * The data model doesn't fit my use case. I know my usage patterns and queries very well, so I am better served by KV stores. * None of the storage engines is a good fit for me. In case you wonder, I've been using RethinkDB and I am now moving to FoundationDB. I need a distributed KV store with strict serializability (https://jepsen.io/consistency/models/strict-serializable https://jepsen.io/consistency/models/strict-serializable) consistency, running on my hardware. In other words, not every database is a good fit for every application. The "just use Postgres!" crowd is wrong, and so is anybody who says "just use Datomic".