4 ms·
Strong agree. there are vast, massive cost savings and performance advantages to be had if the model is that a shard of the dataset is in memory and the data pe
by foobiekr 4y ago
Strong agree. there are vast, massive cost savings and performance advantages to be had if the model is that a shard of the dataset is in memory and the data persistence problem is the part that's made external. The only reason we are where we are today is that doing that well is hard.
- robertlagrant 4y agoIs this not the case already? Database drivers (or just your application code) are allowed to cache results if they like. The problem is cache invalidation.
- foobiekr 4y agoCaches aren't the same. In the shard-in-memory case, the shard is the thing serving the queries, meaning it's not a cache it _is_ the live data.
- robertlagrant 4y agoUnderstood. For a single-node or read-only system it sounds fine, but then there are a variety of ways to solve that (e.g. a preloaded in-memory sqlite).
- bombolo 4y agoIf it's in memory you must live with the fact that it might be gone at any moment though.
- foobiekr 4y agoHence "the data persistence part."
- robertlagrant 4y agoThis sounds like a write-back / write-through cache with extra terminology. What's the difference?
- foobiekr 4y agoThe difference is the 10,000x slower performance if you move the actual DB transaction leader for the shard to over-the-wire access. Anything can be anywhere if we ignore latency and throughput.
- robertlagrant 4y agoI'm not saying that. I'm asking if this is just new terminology for a previous technology idea, or whether it's a new concept.