6 ms·
Datomic's is perfect for probably 90% of small-ish backoffice systems that never has to be web scale (i.e. most of what I do at work). Writing in a single thre
by augustl 3y ago
Datomic's is perfect for probably 90% of small-ish backoffice systems that never has to be web scale (i.e. most of what I do at work).
Writing in a single thread removes a whole host of problems in understanding (and implementing) how data changes over time. (And a busy MVCC sql db spends 75% of its time doing coordination, not actual writes, so a single thread applying a queue of transactions in sequence can be faster than your gut feeling might tell you.)
Transactions as first-class entities of the system means you can easily add meta-data for every change in the system that explains who and why the change happened, so you'll never again have to wonder "hmm, why does that column have that value, and how did it happen". Once you get used to this, doing UPDATE in SQL feels pretty weird, as the default mode of operation of your _business data_ is to delete data, with no trace of who and why!
Having the value of the entire database at a point in time available to your business logic as a (lazy) immutable value you can run queries on opens up completely new ways of writing code, and lets your database follow "functional core, imperative shell". Someone needs to have the working set of your database in memory, why shouldn't it be your app server and business logic?
Looking forward to see what this does for the adoption of Datomic!
- Lutger 3y agoDatomic always seemed like a really cool thing to use. However, I'm not familiar with Clojure or any other JVM based language, nor do I have the time to learn it. And I can't find any supported way to use it with other languages (I'm not even talking about popular frameworks), or am I missing something? It doesn't feel like the people behind Datomic actually want to have users outside of the Clojure world, which will be rather limiting to adoption.
- electroly 3y ago> Someone needs to have the working set of your database in memory, why shouldn't it be your app server and business logic? This one confused me. The obvious reason why you don't want the whole working set of your database in the app server's memory is because you have lots of app servers, whereas you only have one database[1]. This suggests that you put the working set of the database in the database, so that you still only need the one copy, not in the app servers where you'd need N copies of it. The rest of your post makes sense to me but the thing about keeping the database's working set in your app server's memory does not. That's something we specifically work to avoid. [1] Still talking about "non-webscale" office usage here, that's the world I live in as well. One big central database server, lots of apps and app servers strewn about.
- NovemberWhiskey 3y agoI think the point is that treating your database as an arms-length, RPC component that's independent from your "application" isn't necessarily the only pattern.
- foobiekr 3y agoStrong agree. there are vast, massive cost savings and performance advantages to be had if the model is that a shard of the dataset is in memory and the data persistence problem is the part that's made external. The only reason we are where we are today is that doing that well is hard.
- robertlagrant 3y agoIs this not the case already? Database drivers (or just your application code) are allowed to cache results if they like. The problem is cache invalidation.
- foobiekr 3y agoCaches aren't the same. In the shard-in-memory case, the shard is the thing serving the queries, meaning it's not a cache it _is_ the live data.
- robertlagrant 3y agoUnderstood. For a single-node or read-only system it sounds fine, but then there are a variety of ways to solve that (e.g. a preloaded in-memory sqlite).
- bombolo 3y agoIf it's in memory you must live with the fact that it might be gone at any moment though.
- 3y ago
- fulafel 3y ago> Someone needs to have the working set of your database in memory, why shouldn't it be your app server and business logic? This is Ions in the Cloud version, or for on-prem version the in-process peer library.
- Scarbutt 3y agoDatomic's is perfect for probably 90% of small-ish backoffice systems that never has to be web scale How do they scale it for Nubank? (millions of users)
- bhurlow 3y agosharding, microservices, they run many instances of Datomic to handle different functionality
- augustl 3y agoGood question! I don't have any personal experience in that regard. I would probably have paid up for enterprise support (or bought the entire company ;))
- ithrow 3y agoI don't how they do it, but the obvious answer is probably sharding. Their cloud costs must be no joke. Peers require tons of memory and I can only guess they must have thousands of transactors to support that workload and who knows how many peers. Add to this that they probably need something like Kafka for integrating/pipelining all this data.
- outworlder 3y ago> Peers require tons of memory As do most distributed databases. Even when you don't store your entire database (or working set) in memory, you'll likely still have to add quite a bit of memory to be used as I/O cache.
- xpe 3y ago> Datomic's is perfect for probably 90% of small-ish backoffice systems that never has to be web scale (i.e. most of what I do at work). I don’t think I agree with this as stated. It is too squishy and subjective to say “perfect”. More broadly, the above is not and should not be a cognitive “anchor point” for reasonable use cases for Datomic. Making that kind of claim requires a lot more analysis and persuasion.
- augustl 3y agoI agree, I mostly phrased it that way for effect. My "analysis" is 100% subjective, opinionated and anecdotal :)
- pachico 3y agoYou seem to describe the Event Source paradigm rather than a database :)
- augustl 3y agoThe main difference between event sourcing and datomic are the indexes and the "schema", which provides full SQL-like relational query powers out of the box, as well as point-in-time snapshots for every "event" (transactions of facts). So, "events" in Datomic are structured and Datomic uses them to give you query powers, they're not opaque blobs of data.
- epolanski 3y ago> Datomic's is perfect for probably 90% of small-ish backoffice systems that never has to be web scale (i.e. most of what I do at work). So is any cloud-managed db offering and at that scale we talking very small costs anyway. Why datomic instead?
- augustl 3y agoBecause of the reasons I list :) Anything in particular that wasn't clear/relevant?
- brundolf 3y agoSomething I've been curious about: how well (or badly) would it scale to do something similar on a normal relational DB (say, Postgres)? You could have one or more append-only tables that store events/transactions/whatever you want to call them, and then materialized-views (or whatever) which gather that history into a "current state" of "entities", as needed If eventual-consistency is acceptable, it seems like you could aggressively cache and/or distribute reads. Maybe you could even do clever stuff like recomputing state only from the last event you had, instead of from scratch every time How bad of an idea is this?
- augustl 3y agoDatomic already sort of does this :) You configure a storage backend (Datomic does not write to disk directly) which can be dynamodb, riak, or any JDBC database including postgres. You won't get readable data in PG though, as Datomic stores opaque compressed chunks in a key/value structure. The chunks are adressable via the small handful of built-in indexes that Datomic provides for querying, and the indexes are covering, i.e. data is duplicated for each index.
- brundolf 3y agoInteresting! I assumed Datomic was entirely custom Now I'm even more curious if you could skip Datomic and just do something like this directly with a relational DB in production
- spariev 3y agoOne thing which is quite hard to do in Datomic is simple pagination on a large sorted dataset, as one can easily do with LIMIT/OFFSET in MySQL for example. There are solutions for some of the cases, but general case is not solved, as far as I remember (it’s been a while I used it extensively)
- augustl 3y agoIt depends! If you want to lazily walk data, you can read directly from the index (keep in mind, the index = the data = lives in your app), or use the index-pull API which is a bit more convenient. However, if you want to paginate data that you need to sort first, and the data isn't sorted the way you want in the index, you have to read all of the data first, and then sort it. But this is also what a database server would need to do :)
- spariev 3y agoYep, I am well aware of these specifics and workarounds, but in general case where is no general solution to the question asked here, for example [0]. And for big datasets with complex sorting it will take some effort to implement a seemingly simple feature. Guess it is just one of the tradeoffs, as while some features Datomic has out of the box are hard to replicate in RDBMS-es, things like pagination which are often took for granted is a bit of work to do in Datomic. So it is something to keep in mind when considering the switch [0] https://forum.datomic.com/t/idiomatic-pagination-using-latest-features/1454 https://forum.datomic.com/t/idiomatic-pagination-using-lates...
- augustl 3y agoInteresting link, thanks for posting! Datomic's covering indexes are heavily based on their built-in ordering, and doesn't really have much flexibility in different ways to sort and walk data. Personally, I'm a fan of hybrid database approaches. In the world of serverless, I really enjoy the combo of DynamoDB and Elasticsearch, for example, where Dynamo handles everything that's performance critical, and Elasticsearch handles everything where dynamic queries (and ordering, and aggregation, and ....) is required. I've never done this with Datomic, but I'd imagine mirroring the "current" value of entities without historical data is relatively easy to set up.
- JimmyRuska 3y ago> doing UPDATE in SQL feels pretty weird, as the default mode of operation of your _business data_ is to delete data, with no trace of who and why! It's a good idea to version your schema changes using something like liquibase into git, that gets rid of at least some of those pains. Liquibase works on a wide variety of databases, even graphs like Neo4j I got the same feeling in Erlang many times, once write operations start getting parallel you worry about atomic operations, and making an Erlang process centralize writes through its message queue always feels natural and easy to reason about.