3 ms·
At 40:00: Time-based data, horizontally scalable, dumb storage, replicas don't need to be synchronized, you can contact any replica. Sounds like the dream. Bu
by Groxx 7y ago
At 40:00: Time-based data, horizontally scalable, dumb storage, replicas don't need to be synchronized, you can contact any replica. Sounds like the dream.
But that's a drastic over-simplification of time and consistent responses. Consistent responses in particular seems like a hard requirement given how much attention he's given to "why N+1 queries are wrong and you need to do them as a single operation" (which I totally agree with). And I don't think this is something that can be glossed over at all because the whole concept hinges on history being immutable... which means a major synchronization time bottleneck.
And while he does address this at 46:40, but he seems to treat inconsistent timestamps as "this is an interesting space, but if you have [inconsistent time in your db] you have a problem" as if it's a solved problem. Spanner has arguably the most sophisticated approach at scale right now, with atomic clocks all over the place, and it still uses MVCC to address the discrepancies because to do otherwise means major scaling problems.
---
There's a lot to unpack in here, delivered rapidly and concisely (yay!), and some good detailed chunks of info, but also what feels like a lot of "these are wrong and/or difficult for specific and largely-correct reasons" -> "this could be great because [hands waving furiously] it has none of those problems!".
Values are great, documents are terrible, programmability is king, I completely agree. But none of this sounds like it leads to a solution, just a "these are the tradeoffs we're making, how about we stop trading them off?". As a thought-provoker it's interesting, but it doesn't seem to directly lead to anything.
- polityagent 7y agoI believe it leads to datomic, a database you can try out on Aws (datomic cloud) to see the implementation of all the things he's getting excited about.
- dustingetz 7y agoDatomic works, all queries have a time-basis and all results are strongly consistent, full stop. At a mile high view, it works like git. There is a central transactor coordinating time, so inconsistent timestamps is not an issue. To deal with staleness, query nodes connect to the transactor and subscribe to the transaction log. If you send a query with a recent time-basis to a stale query node, it can block until it's caught up enough. The lag time is <10ms for query nodes in same data center as the transactor (and for web applications, normal server response times are already slower than 10ms) Datomic is not for Spanner-scale systems, it is an application database competitive with relational databases.
- Groxx 7y ago>There is a central transactor coordinating time, so inconsistent timestamps is not an issue. This means replicas are synchronized. Or you cannot have replicas. Whether it's a lightweight "transaction-only" replica or not doesn't really matter (I'm under the fuzzy impression that postgres's txids serve this same purpose?).